{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# XGBoost Parameter Tuning for Rent Listing Inqueries Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rental Listing Inquiries数据集是Kaggle平台上的一个分类竞赛任务，需要根据公寓的特征来预测其受欢迎程度（用户感兴趣程度分为高、中、低三类）。其中房屋的特征x共有14维，响应值y为用户对该公寓的感兴趣程度。评价标准为logloss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5.参数调优：subsample & colsample_bytree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用GridSearchCV进行两个参数的同时调优  \n",
    "注：GridSearchCV中最终判定的是scoring越大的模型越好，若用log_loss作为评价指标，在函数中要用neg_log_loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from xgboost import XGBClassifier\n",
    "import xgboost as xgb\n",
    "\n",
    "import pandas as pd \n",
    "import numpy as np\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "\n",
    "from sklearn.metrics import log_loss\n",
    "\n",
    "from matplotlib import pyplot\n",
    "import seaborn as sns\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 读取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>bathrooms</th>\n",
       "      <th>bedrooms</th>\n",
       "      <th>price</th>\n",
       "      <th>price_bathrooms</th>\n",
       "      <th>price_bedrooms</th>\n",
       "      <th>room_diff</th>\n",
       "      <th>room_num</th>\n",
       "      <th>Year</th>\n",
       "      <th>Month</th>\n",
       "      <th>Day</th>\n",
       "      <th>...</th>\n",
       "      <th>walk</th>\n",
       "      <th>walls</th>\n",
       "      <th>war</th>\n",
       "      <th>washer</th>\n",
       "      <th>water</th>\n",
       "      <th>wheelchair</th>\n",
       "      <th>wifi</th>\n",
       "      <th>windows</th>\n",
       "      <th>work</th>\n",
       "      <th>interest_level</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.5</td>\n",
       "      <td>3</td>\n",
       "      <td>3000</td>\n",
       "      <td>1200.0</td>\n",
       "      <td>750.000000</td>\n",
       "      <td>-1.5</td>\n",
       "      <td>4.5</td>\n",
       "      <td>2016</td>\n",
       "      <td>6</td>\n",
       "      <td>24</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>5465</td>\n",
       "      <td>2732.5</td>\n",
       "      <td>1821.666667</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>6</td>\n",
       "      <td>12</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2850</td>\n",
       "      <td>1425.0</td>\n",
       "      <td>1425.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>4</td>\n",
       "      <td>17</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>3275</td>\n",
       "      <td>1637.5</td>\n",
       "      <td>1637.500000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>4</td>\n",
       "      <td>18</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>4</td>\n",
       "      <td>3350</td>\n",
       "      <td>1675.0</td>\n",
       "      <td>670.000000</td>\n",
       "      <td>-3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>4</td>\n",
       "      <td>28</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 228 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   bathrooms  bedrooms  price  price_bathrooms  price_bedrooms  room_diff  \\\n",
       "0        1.5         3   3000           1200.0      750.000000       -1.5   \n",
       "1        1.0         2   5465           2732.5     1821.666667       -1.0   \n",
       "2        1.0         1   2850           1425.0     1425.000000        0.0   \n",
       "3        1.0         1   3275           1637.5     1637.500000        0.0   \n",
       "4        1.0         4   3350           1675.0      670.000000       -3.0   \n",
       "\n",
       "   room_num  Year  Month  Day       ...        walk  walls  war  washer  \\\n",
       "0       4.5  2016      6   24       ...           0      0    0       0   \n",
       "1       3.0  2016      6   12       ...           0      0    0       0   \n",
       "2       2.0  2016      4   17       ...           0      0    0       0   \n",
       "3       2.0  2016      4   18       ...           0      0    0       0   \n",
       "4       5.0  2016      4   28       ...           0      0    1       0   \n",
       "\n",
       "   water  wheelchair  wifi  windows  work  interest_level  \n",
       "0      0           0     0        0     0               1  \n",
       "1      0           0     0        0     0               2  \n",
       "2      0           0     0        0     0               0  \n",
       "3      0           0     0        0     0               2  \n",
       "4      0           0     0        0     0               2  \n",
       "\n",
       "[5 rows x 228 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# path to where the data lies\n",
    "dpath = './data/'\n",
    "train = pd.read_csv(dpath +\"RentListingInquries_FE_train.csv\")\n",
    "test = pd.read_csv(dpath +\"RentListingInquries_FE_test.csv\")\n",
    "train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>bathrooms</th>\n",
       "      <th>bedrooms</th>\n",
       "      <th>price</th>\n",
       "      <th>price_bathrooms</th>\n",
       "      <th>price_bedrooms</th>\n",
       "      <th>room_diff</th>\n",
       "      <th>room_num</th>\n",
       "      <th>Year</th>\n",
       "      <th>Month</th>\n",
       "      <th>Day</th>\n",
       "      <th>...</th>\n",
       "      <th>virtual</th>\n",
       "      <th>walk</th>\n",
       "      <th>walls</th>\n",
       "      <th>war</th>\n",
       "      <th>washer</th>\n",
       "      <th>water</th>\n",
       "      <th>wheelchair</th>\n",
       "      <th>wifi</th>\n",
       "      <th>windows</th>\n",
       "      <th>work</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2950</td>\n",
       "      <td>1475.000000</td>\n",
       "      <td>1475.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>6</td>\n",
       "      <td>11</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>2850</td>\n",
       "      <td>1425.000000</td>\n",
       "      <td>950.000000</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>6</td>\n",
       "      <td>24</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>3758</td>\n",
       "      <td>1879.000000</td>\n",
       "      <td>1879.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>3300</td>\n",
       "      <td>1650.000000</td>\n",
       "      <td>1100.000000</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>6</td>\n",
       "      <td>11</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2.0</td>\n",
       "      <td>2</td>\n",
       "      <td>4900</td>\n",
       "      <td>1633.333333</td>\n",
       "      <td>1633.333333</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>4</td>\n",
       "      <td>12</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 227 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   bathrooms  bedrooms  price  price_bathrooms  price_bedrooms  room_diff  \\\n",
       "0        1.0         1   2950      1475.000000     1475.000000        0.0   \n",
       "1        1.0         2   2850      1425.000000      950.000000       -1.0   \n",
       "2        1.0         1   3758      1879.000000     1879.000000        0.0   \n",
       "3        1.0         2   3300      1650.000000     1100.000000       -1.0   \n",
       "4        2.0         2   4900      1633.333333     1633.333333        0.0   \n",
       "\n",
       "   room_num  Year  Month  Day  ...   virtual  walk  walls  war  washer  water  \\\n",
       "0       2.0  2016      6   11  ...         0     0      0    0       0      0   \n",
       "1       3.0  2016      6   24  ...         0     0      0    1       0      0   \n",
       "2       2.0  2016      6    3  ...         0     0      0    0       0      0   \n",
       "3       3.0  2016      6   11  ...         0     0      0    0       0      0   \n",
       "4       4.0  2016      4   12  ...         0     0      0    1       0      0   \n",
       "\n",
       "   wheelchair  wifi  windows  work  \n",
       "0           0     0        0     0  \n",
       "1           0     0        0     0  \n",
       "2           0     0        0     0  \n",
       "3           1     0        0     0  \n",
       "4           0     0        0     0  \n",
       "\n",
       "[5 rows x 227 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "发现测试集中没有y，这次作业无法得到在测试集上的表现情况"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*********** train **********\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 49352 entries, 0 to 49351\n",
      "Columns: 228 entries, bathrooms to interest_level\n",
      "dtypes: float64(9), int64(219)\n",
      "memory usage: 85.8 MB\n",
      "************ test ***********\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 74659 entries, 0 to 74658\n",
      "Columns: 227 entries, bathrooms to work\n",
      "dtypes: float64(9), int64(218)\n",
      "memory usage: 129.3 MB\n"
     ]
    }
   ],
   "source": [
    "print(\"*********** train **********\")\n",
    "train.info()\n",
    "print(\"************ test ***********\")\n",
    "test.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "测试集中少一维，即y标签"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZUAAAELCAYAAAARNxsIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAGx5JREFUeJzt3X2wHXWd5/H3h/AgPibIhcokYYKaGUUdo14hypSj6EDAHRMpUChHIkNtxAKF0rUEd5aMPMyM4ygrM8pupogkrkOIiBI1GDMMyII8JGB4CJHlioxcyZJgeNQVivDZP/p35eTm5N7OpU9OTu7nVdV1ur/96z7fwy340v379a9lm4iIiCbs0e0EIiJi95GiEhERjUlRiYiIxqSoREREY1JUIiKiMSkqERHRmBSViIhoTIpKREQ0JkUlIiIas2e3E9jZ9t9/f0+fPr3baURE9JTbbrvtEdt9o7Ubd0Vl+vTprFmzpttpRET0FEn/Uaddbn9FRERjOlZUJL1I0q2S7pC0TtLnS/xSSb+QtLYsM0tcki6SNCDpTklvaTnXPEn3lWVeS/ytku4qx1wkSZ36PRERMbpO3v56GjjC9lOS9gJukHR12fcZ21cMa380MKMshwEXA4dJ2g9YAPQDBm6TtNz2o6XNfOBmYAUwG7iaiIjoio5dqbjyVNncqywjzbM/B1hSjrsZmChpMnAUsMr25lJIVgGzy76X277J1fz9S4C5nfo9ERExuo72qUiaIGktsJGqMNxSdl1QbnFdKGmfEpsCPNhy+GCJjRQfbBNvl8d8SWskrdm0adML/l0REdFeR4uK7S22ZwJTgUMlvQE4G3gt8DZgP+CzpXm7/hCPId4uj4W2+2339/WNOiIuIiLGaKeM/rL9GHAdMNv2hnKL62ng68ChpdkgMK3lsKnAQ6PEp7aJR0REl3Ry9FefpIllfV/gvcDPSl8IZaTWXODucshy4KQyCmwW8LjtDcBK4EhJkyRNAo4EVpZ9T0qaVc51EnBVp35PRESMrpOjvyYDiyVNoCpey2x/X9K/S+qjun21Fji1tF8BHAMMAL8FTgawvVnSecDq0u5c25vL+seBS4F9qUZ9ZeRXREQXqRo4NX709/c7T9RH7NoO/6fDu53Cbu/GT9y4Q+0l3Wa7f7R2eaI+IiIak6ISERGNSVGJiIjGpKhERERjUlQiIqIxKSoREdGYFJWIiGhMikpERDQmRSUiIhqTohIREY1JUYmIiMakqERERGNSVCIiojEpKhER0ZgUlYiIaEyKSkRENCZFJSIiGpOiEhERjUlRiYiIxqSoREREY1JUIiKiMR0rKpJeJOlWSXdIWifp8yV+sKRbJN0n6XJJe5f4PmV7oOyf3nKus0v8XklHtcRnl9iApLM69VsiIqKeTl6pPA0cYftNwExgtqRZwBeAC23PAB4FTintTwEetf0a4MLSDkmHACcArwdmA1+TNEHSBOCrwNHAIcCJpW1ERHRJx4qKK0+Vzb3KYuAI4IoSXwzMLetzyjZl/3skqcSX2n7a9i+AAeDQsgzYvt/2M8DS0jYiIrqko30q5YpiLbARWAX8HHjM9rOlySAwpaxPAR4EKPsfB17ZGh92zPbiERHRJR0tKra32J4JTKW6snhdu2blU9vZt6PxbUiaL2mNpDWbNm0aPfGIiBiTnTL6y/ZjwHXALGCipD3LrqnAQ2V9EJgGUPa/AtjcGh92zPbi7b5/oe1+2/19fX1N/KSIiGijk6O/+iRNLOv7Au8F1gPXAseVZvOAq8r68rJN2f/vtl3iJ5TRYQcDM4BbgdXAjDKabG+qzvzlnfo9ERExuj1HbzJmk4HFZZTWHsAy29+XdA+wVNL5wE+BS0r7S4BvSBqgukI5AcD2OknLgHuAZ4HTbG8BkHQ6sBKYACyyva6DvyciIkbRsaJi+07gzW3i91P1rwyP/w44fjvnugC4oE18BbDiBScbERGNyBP1ERHRmBSViIhoTIpKREQ0JkUlIiIak6ISERGNSVGJiIjGpKhERERjUlQiIqIxKSoREdGYFJWIiGjMqEVF0ksk7VHW/0jS+yXt1fnUIiKi19S5UrkeeJGkKcA1wMnApZ1MKiIielOdoiLbvwWOBf7J9geo3gkfERGxlVpFRdLbgQ8DPyixTk6ZHxERPapOUTkTOBv4Tnm3yauoXrQVERGxlVGvOGz/GPixpJeU7fuBT3Y6sYiI6D11Rn+9vbytcX3ZfpOkr3U8s4iI6Dl1bn/9d+Ao4NcAtu8A3tnJpCIiojfVevjR9oPDQls6kEtERPS4OqO4HpT0DsCS9qbqT1nf2bQiIqIX1blSORU4DZgCDAIzy3ZERMRWRi0qth+x/WHbB9o+wPZf2v71aMdJmibpWknrJa2TdEaJ/42kX0laW5ZjWo45W9KApHslHdUSn11iA5LOaokfLOkWSfdJurxcSUVERJfUGf21WNLElu1JkhbVOPezwKdtvw6YBZwmaehJ/AttzyzLinLeQ4ATgNcDs4GvSZogaQLwVeBoqif5T2w5zxfKuWYAjwKn1MgrIiI6pM7trz+x/djQhu1HgTePdpDtDbZvL+tPUvXDTBnhkDnAUttP2/4FMAAcWpYB2/fbfgZYCsyRJOAI4Ipy/GJgbo3fExERHVKnqOwhadLQhqT92MFpWiRNpypEt5TQ6ZLulLSo5dxTgNZRZoMltr34K4HHbD87LB4REV1Sp6h8CfiJpPMknQf8BPiHul8g6aXAt4EzbT8BXAy8mqrDf0M5P4DaHO4xxNvlMF/SGklrNm3aVDf1iIjYQXU66pcAxwEPAxuBY21/o87Jy3tXvg180/aV5XwP295i+zngX6hub0F1pTGt5fCpwEMjxB8BJkrac1i83W9YaLvfdn9fX1+d1CMiYgzqvvnxZ8CVwFXAU5IOGu2A0udxCbDe9pdb4pNbmn0AuLusLwdOkLSPpIOBGcCtwGpgRhnptTdVZ/5y26aa2PK4cvy8kl9ERHTJqH0jkj4BLKC6UtlCddvJwJ+McujhwEeAuyStLbHPUY3emlnO8QDwMYAyA/Iy4B6qkWOn2d5ScjgdWAlMABbZXlfO91lgqaTzgZ9SFbGIiOiSOh3uZwB/XOfZlFa2b6B9v8eKEY65ALigTXxFu+PKjMmHDo9HRER31Ln99SDweKcTiYiI3lfnSuV+4DpJPwCeHgq29pNERERAvaLyy7LsXZaIiIi26rz58fMAkl5i+zedTykiInpV3vwYERGNyZsfIyKiMXnzY0RENCZvfoyIiMbkzY8REdGYEa9UyguyPmL7wzspn4iI6GEjXqmUubfm7KRcIiKix9XpU7lR0j8DlwO/f05l6K2OERERQ+oUlXeUz3NbYqZ6lW9ERMTvjdansgdwse1lOymfiIjoYaP1qTwHnL6TcomIiB5XZ0jxKkn/RdI0SfsNLR3PLCIiek6dPpW/Kp+tz6YYeFXz6URERC+rM0vxwTsjkYiI6H113lF/Uru47SXNpxMREb2szu2vt7Wsvwh4D3A7kKISERFbqXP76xOt25JeAXyjYxlFRETPqjX1/TC/BWaM1qiMFrtW0npJ6ySdUeL7SVol6b7yOanEJekiSQOS7pT0lpZzzSvt75M0ryX+Vkl3lWMukqQx/J6IiGhInTc/fk/S8rJ8H7gXuKrGuZ8FPm37dcAs4DRJhwBnAdfYngFcU7YBjqYqVjOA+cDF5fv3AxYAhwGHAguGClFpM7/luNk18oqIiA6p06fyjy3rzwL/YXtwtINsbwA2lPUnJa2nmj5/DvCu0mwxcB3w2RJfYtvAzZImSppc2q6yvRlA0ipgtqTrgJfbvqnElwBzgatr/KaIiOiAOkXll8AG278DkLSvpOm2H6j7JZKmA28GbgEOLAUH2xskHVCaTQFa3zA5WGIjxQfbxCMiokvq9Kl8C3iuZXtLidUi6aXAt4EzbT8xUtM2MY8h3i6H+ZLWSFqzadOm0VKOiIgxqlNU9rT9zNBGWd+7zskl7UVVUL5p+8oSfrjc1qJ8bizxQWBay+FTgYdGiU9tE9+G7YW2+2339/X11Uk9IiLGoE5R2STp/UMbkuYAj4x2UBmJdQmw3vaXW3YtB4ZGcM3j+U7/5cBJZRTYLODxcptsJXCkpEmlg/5IYGXZ96SkWeW7TqLeAIKIiOiQOn0qpwLfLC/qguoKoe1T9sMcDnwEuEvS2hL7HPD3wDJJp1D11xxf9q0AjgEGqIYtnwxge7Ok84DVpd25Q532wMeBS4F9qTro00kfEdFFdR5+/Dkwq/SNyPaTdU5s+wba93tA9VT+8PZm60krW/ctAha1ia8B3lAnn4iI6Lw6z6n8raSJtp8qQ4MnSTp/ZyQXERG9pU6fytG2HxvasP0o1W2qiIiIrdQpKhMk7TO0IWlfYJ8R2kdExDhVp6P+fwHXSPo61XMgf0X1JHxERMRW6nTU/4OkO4H3ltB5tld2Nq2IiOhFda5UAH4K7EV1pfLTzqUTERG9rM7orw8CtwLHAR8EbpF0XKcTi4iI3lPnSuW/Am+zvRFAUh/wb8AVnUwsIiJ6T53RX3sMFZTi1zWPi4iIcabOlcoPJa0ELivbH6KaUiUiImIrdUZ/fUbSscCfUk27stD2dzqeWURE9Jxao7/KtPVXjtowIiLGtfSNREREY1JUIiKiMdstKpKuKZ9f2HnpRERELxupT2WypD8D3i9pKcPejWL79o5mFhERPWekonIOcBbVu9+/PGyfgSM6lVRERPSm7RYV21cAV0j6b7bP24k5RUREj6rznMp5kt4PvLOErrP9/c6mFRERvajOhJJ/B5wB3FOWM0osIiJiK3UefnwfMNP2cwCSFlNNf392JxOLiIjeU/c5lYkt66/oRCIREdH76hSVvwN+KunScpVyG/C3ox0kaZGkjZLubon9jaRfSVpblmNa9p0taUDSvZKOaonPLrEBSWe1xA+WdIuk+yRdLmnvuj86IiI6Y9SiYvsyYBbV3F9XAm+3vbTGuS8FZreJX2h7ZllWAEg6BDgBeH055muSJkiaAHwVOBo4BDixtAX4QjnXDOBR4JQaOUVERAfVuv1le4Pt5bavsv1/ax5zPbC5Zh5zgKW2n7b9C2AAOLQsA7bvt/0MsBSYI0lUz8kMvShsMTC35ndFRESHdGPur9Ml3Vluj00qsSnAgy1tBktse/FXAo/ZfnZYvC1J8yWtkbRm06ZNTf2OiIgYZmcXlYuBVwMzgQ3Al0pcbdp6DPG2bC+03W+7v6+vb8cyjoiI2kYsKpL2aO1of6FsP2x7Sxme/C9Ut7egutKY1tJ0KvDQCPFHgImS9hwWj4iILhqxqJT/+N8h6aAmvkzS5JbNDwBDBWs5cIKkfSQdDMwAbgVWAzPKSK+9qTrzl9s2cC1wXDl+HnBVEzlGRMTY1Xn4cTKwTtKtwG+GgrbfP9JBki4D3gXsL2kQWAC8S9JMqltVDwAfK+daJ2kZ1RP7zwKn2d5SznM6sBKYACyyva58xWeBpZLOp3oY85I6PzgiIjqnTlH5/FhObPvENuHt/off9gXABW3iK4AVbeL38/zts4iI2AXUmVDyx5L+EJhh+98kvZjqqiEiImIrdSaU/M9Uz4P8zxKaAny3k0lFRERvqjOk+DTgcOAJANv3AQd0MqmIiOhNdYrK0+VpdgDKMN7tPhMSERHjV52i8mNJnwP2lfTnwLeA73U2rYiI6EV1ispZwCbgLqohwCuAv+5kUhER0ZvqjP56rkx5fwvVba97y8OHERERWxm1qEh6H/A/gJ9Tzbl1sKSP2b6608lFRERvqfPw45eAd9seAJD0auAHQIpKRERspU6fysahglLcD2zsUD4REdHDtnulIunYsrpO0gpgGVWfyvFUEz1GRERsZaTbX3/Rsv4w8GdlfRMwadvmEREx3m23qNg+eWcmEhERva/O6K+DgU8A01vbjzb1fUREjD91Rn99l2rK+u8Bz3U2nYiI6GV1isrvbF/U8UwiIqLn1SkqX5G0APgR8PRQ0PbtHcsqIiJ6Up2i8kbgI8ARPH/7y2U7Ypf0y3Pf2O0UxoWDzrmr2ynELqZOUfkA8KrW6e8jIiLaqfNE/R3AxE4nEhERva/OlcqBwM8krWbrPpUMKY6IiK3UKSoLxnJiSYuA/0Q1d9gbSmw/4HKqZ14eAD5o+1FJAr4CHAP8Fvjo0EAASfN4/v0t59teXOJvBS4F9qV6x8sZmZI/IqK7Rr39ZfvH7ZYa574UmD0sdhZwje0ZwDVlG+BoYEZZ5gMXw++L0ALgMOBQYIGkoSliLi5th44b/l0REbGTjVpUJD0p6Ymy/E7SFklPjHac7euBzcPCc4DFZX0xMLclvsSVm4GJkiYDRwGrbG+2/SiwCphd9r3c9k3l6mRJy7kiIqJL6rz58WWt25LmUl01jMWBtjeU826QdECJTwEebGk3WGIjxQfbxNuSNJ/qqoaDDjpojKlHRMRo6oz+2ort79L8Mypq91VjiLdle6Htftv9fX19Y0wxIiJGU2dCyWNbNvcA+hnhP+CjeFjS5HKVMpnnX/Y1CExraTcVeKjE3zUsfl2JT23TPiIiuqjOlcpftCxHAU9S9YGMxXJgXlmfB1zVEj9JlVnA4+U22UrgSEmTSgf9kcDKsu9JSbPKyLGTWs4VERFdUqdPZUzvVZF0GdVVxv6SBqlGcf09sEzSKcAvqd4iCdWQ4GOAAaohxSeX794s6Tyef9PkubaHOv8/zvNDiq8uS0REdNFIrxM+Z4TjbPu8kU5s+8Tt7HpPu5MBp23nPIuARW3ia4A3jJRDRETsXCNdqfymTewlwCnAK4ERi0pERIw/I71O+EtD65JeBpxBdVtqKfCl7R0XERHj14h9KuWJ9k8BH6Z6WPEt5SHEiIiIbYzUp/JF4FhgIfBG20/ttKwiIqInjTSk+NPAH1BN5vhQy1QtT9aZpiUiIsafkfpUdvhp+4iIGN9SOCIiojEpKhER0ZgUlYiIaEyKSkRENCZFJSIiGpOiEhERjUlRiYiIxqSoREREY1JUIiKiMSkqERHRmBSViIhoTIpKREQ0JkUlIiIak6ISERGNSVGJiIjGdKWoSHpA0l2S1kpaU2L7SVol6b7yOanEJekiSQOS7pT0lpbzzCvt75M0rxu/JSIintfNK5V3255pu79snwVcY3sGcE3ZBjgamFGW+cDFUBUhYAFwGHAosGCoEEVERHfsSre/5gCLy/piYG5LfIkrNwMTJU0GjgJW2d5s+1FgFTB7ZycdERHP61ZRMfAjSbdJml9iB9reAFA+DyjxKcCDLccOltj24hER0SXbfUd9hx1u+yFJBwCrJP1shLZqE/MI8W1PUBWu+QAHHXTQjuYaERE1deVKxfZD5XMj8B2qPpGHy20tyufG0nwQmNZy+FTgoRHi7b5voe1+2/19fX1N/pSIiGix04uKpJdIetnQOnAkcDewHBgawTUPuKqsLwdOKqPAZgGPl9tjK4EjJU0qHfRHllhERHRJN25/HQh8R9LQ9/+r7R9KWg0sk3QK8Evg+NJ+BXAMMAD8FjgZwPZmSecBq0u7c21v3nk/IyIihtvpRcX2/cCb2sR/DbynTdzAads51yJgUdM5RkTE2OxKQ4ojIqLHpahERERjujWkuCe89TNLup3Cbu+2L57U7RQiokG5UomIiMakqERERGNSVCIiojEpKhER0ZgUlYiIaEyKSkRENCZFJSIiGpOiEhERjUlRiYiIxqSoREREY1JUIiKiMSkqERHRmBSViIhoTIpKREQ0JkUlIiIak6ISERGNSVGJiIjGpKhERERjer6oSJot6V5JA5LO6nY+ERHjWU8XFUkTgK8CRwOHACdKOqS7WUVEjF89XVSAQ4EB2/fbfgZYCszpck4REeNWrxeVKcCDLduDJRYREV2wZ7cTeIHUJuZtGknzgfll8ylJ93Y0q+7aH3ik20nUpX+c1+0UdiU99bcDYEG7fwXHrZ76++mTO/y3+8M6jXq9qAwC01q2pwIPDW9keyGwcGcl1U2S1tju73YesePyt+tt+ftVev3212pghqSDJe0NnAAs73JOERHjVk9fqdh+VtLpwEpgArDI9roupxURMW71dFEBsL0CWNHtPHYh4+I2324qf7velr8fIHubfu2IiIgx6fU+lYiI2IWkqOwmMl1N75K0SNJGSXd3O5fYMZKmSbpW0npJ6ySd0e2cui23v3YDZbqa/wP8OdUw69XAibbv6WpiUYukdwJPAUtsv6Hb+UR9kiYDk23fLullwG3A3PH8716uVHYPma6mh9m+Htjc7Txix9neYPv2sv4ksJ5xPqtHisruIdPVRHSZpOnAm4FbuptJd6Wo7B5qTVcTEZ0h6aXAt4EzbT/R7Xy6KUVl91BrupqIaJ6kvagKyjdtX9ntfLotRWX3kOlqIrpAkoBLgPW2v9ztfHYFKSq7AdvPAkPT1awHlmW6mt4h6TLgJuCPJQ1KOqXbOUVthwMfAY6QtLYsx3Q7qW7KkOKIiGhMrlQiIqIxKSoREdGYFJWIiGhMikpERDQmRSUiIhqTohIREY1JUYkAJP2kRpszJb24w3nMHO05B0kflfTPDX9v4+eM8SlFJQKw/Y4azc4EdqiolNcS7IiZwLh+eC56W4pKBCDpqfL5LknXSbpC0s8kfVOVTwJ/AFwr6drS9khJN0m6XdK3yqSCSHpA0jmSbgCOl/RqST+UdJuk/y3ptaXd8ZLulnSHpOvLFDvnAh8qT2Z/qEbefZK+LWl1WQ6XtEfJYWJLuwFJB7Zr3/g/zBjX9ux2AhG7oDcDr6ealPNG4HDbF0n6FPBu249I2h/4a+C9tn8j6bPAp6iKAsDvbP8pgKRrgFNt3yfpMOBrwBHAOcBRtn8laaLtZySdA/TbPr1mrl8BLrR9g6SDgJW2XyfpKuADwNfLdz5g+2FJ/zq8PfC6F/jPK+L3UlQitnWr7UEASWuB6cANw9rMAg4BbqzmFGRvqvm7hlxejn8p8A7gW6UdwD7l80bgUknLgLHObvte4JCWc7+8vIHwcqqi9XWqCUYvH6V9RCNSVCK29XTL+hba/3siYJXtE7dzjt+Uzz2Ax2zPHN7A9qnlKuJ9wFpJ27SpYQ/g7bb/31bJSTcBr5HUB8wFzh+l/Ri+OmJb6VOJqO9JYOj/6m8GDpf0GgBJL5b0R8MPKC9s+oWk40s7SXpTWX+17VtsnwM8QvVOnNbvqONHVDNUU845s3yvge8AX6aalv3XI7WPaEqKSkR9C4GrJV1rexPwUeAySXdSFZnXbue4DwOnSLoDWAfMKfEvSrpL0t3A9cAdwLVUt6dqddQDnwT6Jd0p6R7g1JZ9lwN/yfO3vkZrH/GCZer7iIhoTK5UIiKiMemoj9hFSToZOGNY+Ebbp3Ujn4g6cvsrIiIak9tfERHRmBSViIhoTIpKREQ0JkUlIiIak6ISERGN+f+UQGoL+CmGNwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.countplot(train.interest_level);\n",
    "pyplot.xlabel('interest_level');\n",
    "pyplot.ylabel('Number of occurrences');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "总共有三类样本，每类样本分布的并不均匀，一般类别之间的数量差别在十倍以上时才考虑使用class_weight，这里暂不考虑"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop ids and get labels\n",
    "y_train = train['interest_level']\n",
    "\n",
    "train = train.drop([\"interest_level\"], axis=1)\n",
    "X_train = np.array(train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prepare cross validation\n",
    "kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用前几轮调好的参数：  \n",
    "n_estimator = 338  \n",
    "max_depth = 5  \n",
    "min_child_weight = 2  \n",
    "调节subsample和colsample_bytree，其他参数缺省"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'colsample_bytree': [0.6, 0.7, 0.8, 0.9],\n",
       " 'subsample': [0.3, 0.4, 0.5, 0.6, 0.7, 0.8]}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#max_depth 建议3-10， min_child_weight=1／sqrt(ratio_rare_event) =5.5\n",
    "subsample = [i/10.0 for i in range(3,9)]\n",
    "colsample_bytree = [i/10.0 for i in range(6,10)]\n",
    "param_test3_1 = dict(subsample=subsample, colsample_bytree=colsample_bytree)\n",
    "param_test3_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\model_selection\\_search.py:761: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "([mean: -0.58714, std: 0.00375, params: {'colsample_bytree': 0.6, 'subsample': 0.3},\n",
       "  mean: -0.58505, std: 0.00384, params: {'colsample_bytree': 0.6, 'subsample': 0.4},\n",
       "  mean: -0.58327, std: 0.00310, params: {'colsample_bytree': 0.6, 'subsample': 0.5},\n",
       "  mean: -0.58268, std: 0.00326, params: {'colsample_bytree': 0.6, 'subsample': 0.6},\n",
       "  mean: -0.58216, std: 0.00379, params: {'colsample_bytree': 0.6, 'subsample': 0.7},\n",
       "  mean: -0.58143, std: 0.00330, params: {'colsample_bytree': 0.6, 'subsample': 0.8},\n",
       "  mean: -0.58736, std: 0.00439, params: {'colsample_bytree': 0.7, 'subsample': 0.3},\n",
       "  mean: -0.58500, std: 0.00368, params: {'colsample_bytree': 0.7, 'subsample': 0.4},\n",
       "  mean: -0.58365, std: 0.00432, params: {'colsample_bytree': 0.7, 'subsample': 0.5},\n",
       "  mean: -0.58296, std: 0.00387, params: {'colsample_bytree': 0.7, 'subsample': 0.6},\n",
       "  mean: -0.58223, std: 0.00414, params: {'colsample_bytree': 0.7, 'subsample': 0.7},\n",
       "  mean: -0.58178, std: 0.00387, params: {'colsample_bytree': 0.7, 'subsample': 0.8},\n",
       "  mean: -0.58634, std: 0.00399, params: {'colsample_bytree': 0.8, 'subsample': 0.3},\n",
       "  mean: -0.58523, std: 0.00437, params: {'colsample_bytree': 0.8, 'subsample': 0.4},\n",
       "  mean: -0.58402, std: 0.00385, params: {'colsample_bytree': 0.8, 'subsample': 0.5},\n",
       "  mean: -0.58329, std: 0.00438, params: {'colsample_bytree': 0.8, 'subsample': 0.6},\n",
       "  mean: -0.58231, std: 0.00392, params: {'colsample_bytree': 0.8, 'subsample': 0.7},\n",
       "  mean: -0.58202, std: 0.00372, params: {'colsample_bytree': 0.8, 'subsample': 0.8},\n",
       "  mean: -0.58609, std: 0.00362, params: {'colsample_bytree': 0.9, 'subsample': 0.3},\n",
       "  mean: -0.58484, std: 0.00329, params: {'colsample_bytree': 0.9, 'subsample': 0.4},\n",
       "  mean: -0.58373, std: 0.00452, params: {'colsample_bytree': 0.9, 'subsample': 0.5},\n",
       "  mean: -0.58217, std: 0.00377, params: {'colsample_bytree': 0.9, 'subsample': 0.6},\n",
       "  mean: -0.58223, std: 0.00373, params: {'colsample_bytree': 0.9, 'subsample': 0.7},\n",
       "  mean: -0.58114, std: 0.00391, params: {'colsample_bytree': 0.9, 'subsample': 0.8}],\n",
       " {'colsample_bytree': 0.9, 'subsample': 0.8},\n",
       " -0.5811398453745635)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgb3_1 = XGBClassifier(\n",
    "        learning_rate =0.09, #上一轮尝试了0.01，跑的实在太慢了，还是设置为0.09好了\n",
    "        n_estimators=338,  #第二轮参数调整得到的n_estimators最优值\n",
    "        max_depth=5,\n",
    "        min_child_weight=2,\n",
    "        gamma=0,\n",
    "        subsample=0.5,\n",
    "        colsample_bytree=0.8,\n",
    "        colsample_bylevel = 0.7,\n",
    "        objective= 'multi:softprob',\n",
    "        seed=3)\n",
    "\n",
    "\n",
    "gsearch3_1 = GridSearchCV(xgb3_1, param_grid = param_test3_1, scoring='neg_log_loss',n_jobs=-1, cv=kfold)\n",
    "gsearch3_1.fit(X_train , y_train)\n",
    "\n",
    "gsearch3_1.grid_scores_, gsearch3_1.best_params_,     gsearch3_1.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split4_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'mean_fit_time': array([369.04779115, 422.69851346, 466.12788768, 481.66769681,\n",
       "        485.07655473, 493.4927896 , 403.43381157, 471.18275151,\n",
       "        519.59233847, 543.20417552, 548.91817622, 560.92892885,\n",
       "        451.53086653, 527.59551024, 590.9810678 , 612.84078989,\n",
       "        615.5344914 , 621.45721178, 499.7589396 , 585.21958733,\n",
       "        649.89978538, 676.71266775, 685.72357655, 646.7051446 ]),\n",
       " 'mean_score_time': array([1.56763725, 1.34756889, 1.44755802, 1.3665051 , 1.38891907,\n",
       "        1.40953355, 1.39912643, 1.38091416, 1.43615146, 1.4809804 ,\n",
       "        1.31807265, 1.40152764, 1.54482269, 1.49558983, 1.51980581,\n",
       "        1.41173429, 1.56043253, 1.41113458, 1.53521633, 1.42534323,\n",
       "        1.50359564, 1.51820512, 1.35249562, 1.17517757]),\n",
       " 'mean_test_score': array([-0.58713955, -0.58505464, -0.58327285, -0.58268306, -0.58216243,\n",
       "        -0.58143039, -0.58736127, -0.58499794, -0.58365012, -0.58296055,\n",
       "        -0.58223353, -0.58177869, -0.58633508, -0.58523326, -0.58402303,\n",
       "        -0.58328654, -0.58230552, -0.58201574, -0.58608989, -0.58483525,\n",
       "        -0.58372564, -0.58217165, -0.58222693, -0.58113985]),\n",
       " 'mean_train_score': array([-0.49739077, -0.49379134, -0.49148891, -0.49073426, -0.49065022,\n",
       "        -0.49130982, -0.49426987, -0.49028558, -0.48952018, -0.48853942,\n",
       "        -0.48866269, -0.48904935, -0.49145475, -0.48730519, -0.4854684 ,\n",
       "        -0.48447725, -0.48570102, -0.48644851, -0.4888135 , -0.4847027 ,\n",
       "        -0.48308724, -0.48249303, -0.48317065, -0.4843744 ]),\n",
       " 'param_colsample_bytree': masked_array(data=[0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.7, 0.7, 0.7, 0.7, 0.7,\n",
       "                    0.7, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.9, 0.9, 0.9, 0.9,\n",
       "                    0.9, 0.9],\n",
       "              mask=[False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'param_subsample': masked_array(data=[0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.3, 0.4, 0.5, 0.6, 0.7,\n",
       "                    0.8, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.3, 0.4, 0.5, 0.6,\n",
       "                    0.7, 0.8],\n",
       "              mask=[False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'params': [{'colsample_bytree': 0.6, 'subsample': 0.3},\n",
       "  {'colsample_bytree': 0.6, 'subsample': 0.4},\n",
       "  {'colsample_bytree': 0.6, 'subsample': 0.5},\n",
       "  {'colsample_bytree': 0.6, 'subsample': 0.6},\n",
       "  {'colsample_bytree': 0.6, 'subsample': 0.7},\n",
       "  {'colsample_bytree': 0.6, 'subsample': 0.8},\n",
       "  {'colsample_bytree': 0.7, 'subsample': 0.3},\n",
       "  {'colsample_bytree': 0.7, 'subsample': 0.4},\n",
       "  {'colsample_bytree': 0.7, 'subsample': 0.5},\n",
       "  {'colsample_bytree': 0.7, 'subsample': 0.6},\n",
       "  {'colsample_bytree': 0.7, 'subsample': 0.7},\n",
       "  {'colsample_bytree': 0.7, 'subsample': 0.8},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.3},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.4},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.5},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.6},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.7},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.8},\n",
       "  {'colsample_bytree': 0.9, 'subsample': 0.3},\n",
       "  {'colsample_bytree': 0.9, 'subsample': 0.4},\n",
       "  {'colsample_bytree': 0.9, 'subsample': 0.5},\n",
       "  {'colsample_bytree': 0.9, 'subsample': 0.6},\n",
       "  {'colsample_bytree': 0.9, 'subsample': 0.7},\n",
       "  {'colsample_bytree': 0.9, 'subsample': 0.8}],\n",
       " 'rank_test_score': array([23, 19, 12, 10,  5,  2, 24, 18, 14, 11,  8,  3, 22, 20, 16, 13,  9,\n",
       "         4, 21, 17, 15,  6,  7,  1]),\n",
       " 'split0_test_score': array([-0.58060635, -0.57893482, -0.57765931, -0.57686015, -0.57553552,\n",
       "        -0.57605351, -0.5812477 , -0.57893729, -0.57657142, -0.57593807,\n",
       "        -0.57511342, -0.5750844 , -0.57903903, -0.57742302, -0.57750112,\n",
       "        -0.57562011, -0.57580957, -0.57550221, -0.58087776, -0.57981876,\n",
       "        -0.57664053, -0.57571193, -0.57635661, -0.57524395]),\n",
       " 'split0_train_score': array([-0.49838927, -0.49409109, -0.492271  , -0.49237593, -0.49170724,\n",
       "        -0.49304648, -0.49543144, -0.49301238, -0.48965776, -0.48976272,\n",
       "        -0.49038462, -0.49053347, -0.49382853, -0.48932147, -0.48652086,\n",
       "        -0.48534517, -0.48746692, -0.48699243, -0.49007313, -0.485664  ,\n",
       "        -0.48471631, -0.48264894, -0.48412646, -0.48503382]),\n",
       " 'split1_test_score': array([-0.58620598, -0.58377268, -0.58265523, -0.58147686, -0.58132787,\n",
       "        -0.57965377, -0.58378352, -0.5833323 , -0.58330661, -0.58210459,\n",
       "        -0.58021062, -0.58015522, -0.58648639, -0.58480813, -0.58398141,\n",
       "        -0.58268999, -0.58101719, -0.58052717, -0.58445383, -0.58354487,\n",
       "        -0.58045328, -0.58043435, -0.58053967, -0.57902176]),\n",
       " 'split1_train_score': array([-0.49881443, -0.49524064, -0.49197769, -0.49138331, -0.49119478,\n",
       "        -0.49030258, -0.49432986, -0.4901461 , -0.4897393 , -0.48868335,\n",
       "        -0.48762136, -0.48899505, -0.49133337, -0.48734608, -0.48695605,\n",
       "        -0.48467572, -0.48520226, -0.48658882, -0.48962923, -0.4848802 ,\n",
       "        -0.48365335, -0.48308805, -0.4830557 , -0.48596567]),\n",
       " 'split2_test_score': array([-0.58816199, -0.58423874, -0.58386372, -0.5845083 , -0.58223424,\n",
       "        -0.58195617, -0.58813352, -0.58558   , -0.58226185, -0.58438977,\n",
       "        -0.58394829, -0.58286129, -0.58628841, -0.58544705, -0.58299045,\n",
       "        -0.58302155, -0.58210021, -0.58342831, -0.58526061, -0.58456689,\n",
       "        -0.58580839, -0.58330138, -0.5818176 , -0.58071835]),\n",
       " 'split2_train_score': array([-0.49674112, -0.49369366, -0.49099327, -0.48965706, -0.4902278 ,\n",
       "        -0.49140538, -0.4936808 , -0.49025605, -0.48893476, -0.48848618,\n",
       "        -0.48905805, -0.48825511, -0.4907855 , -0.48771929, -0.48493544,\n",
       "        -0.48474093, -0.48555737, -0.4854813 , -0.48848789, -0.48514401,\n",
       "        -0.48237227, -0.48310092, -0.48290991, -0.483565  ]),\n",
       " 'split3_test_score': array([-0.58880273, -0.58946124, -0.58593936, -0.58460754, -0.58582852,\n",
       "        -0.58473668, -0.59010079, -0.58745053, -0.58702364, -0.58514281,\n",
       "        -0.5862457 , -0.58531175, -0.59036059, -0.58822598, -0.58690396,\n",
       "        -0.58720539, -0.58559178, -0.58466945, -0.58824336, -0.58647219,\n",
       "        -0.58666306, -0.58585195, -0.58592313, -0.58452828]),\n",
       " 'split3_train_score': array([-0.49641012, -0.49328835, -0.49074673, -0.48985309, -0.49052589,\n",
       "        -0.49088584, -0.49529554, -0.48911119, -0.48915379, -0.48873789,\n",
       "        -0.48850793, -0.48938975, -0.49232996, -0.48665641, -0.48560408,\n",
       "        -0.48409919, -0.48512645, -0.48678324, -0.48868636, -0.48341333,\n",
       "        -0.48243031, -0.48186532, -0.48242194, -0.48413625]),\n",
       " 'split4_test_score': array([-0.59192218, -0.58886686, -0.58624751, -0.58596345, -0.58588712,\n",
       "        -0.58475284, -0.59354271, -0.58969102, -0.58908873, -0.58722879,\n",
       "        -0.58565065, -0.58548193, -0.58950196, -0.59026367, -0.58873967,\n",
       "        -0.58789707, -0.58701026, -0.58595275, -0.59161557, -0.58977503,\n",
       "        -0.58906458, -0.58555966, -0.58649892, -0.58618842]),\n",
       " 'split4_train_score': array([-0.49659892, -0.49264298, -0.49145589, -0.49040191, -0.48959537,\n",
       "        -0.49090883, -0.49261172, -0.4889022 , -0.49011531, -0.48702694,\n",
       "        -0.48774152, -0.48807337, -0.48899641, -0.4854827 , -0.48332557,\n",
       "        -0.48352525, -0.48515209, -0.48639677, -0.4871909 , -0.48441194,\n",
       "        -0.48226399, -0.4817619 , -0.48333927, -0.48317125]),\n",
       " 'std_fit_time': array([ 4.20957197,  3.99790628,  3.26934701,  5.39497532,  7.11591117,\n",
       "         5.30250095,  5.62723742,  4.35670389,  1.10322089,  7.72101132,\n",
       "         6.30734884,  2.58207182,  3.42207288,  3.9018444 ,  6.02246177,\n",
       "         4.72771467,  4.97299259,  4.0172904 ,  2.59767152,  7.60757952,\n",
       "         3.47214144,  4.73509213,  6.29716483, 54.10120623]),\n",
       " 'std_score_time': array([0.19089575, 0.01372674, 0.15879018, 0.07757462, 0.09964515,\n",
       "        0.15117086, 0.01767944, 0.02450344, 0.15094488, 0.17777582,\n",
       "        0.01947989, 0.11507972, 0.16072462, 0.16589336, 0.1574767 ,\n",
       "        0.10177562, 0.15165521, 0.07421494, 0.12700316, 0.07427781,\n",
       "        0.17228321, 0.19960488, 0.08025938, 0.16040516]),\n",
       " 'std_test_score': array([0.00374798, 0.0038396 , 0.00310488, 0.00326059, 0.00379292,\n",
       "        0.00329587, 0.00439276, 0.00368462, 0.0043153 , 0.00387478,\n",
       "        0.00413532, 0.00386828, 0.00398731, 0.00437003, 0.00385091,\n",
       "        0.00437753, 0.00391998, 0.00371965, 0.0036248 , 0.0032857 ,\n",
       "        0.00452494, 0.00376952, 0.0037252 , 0.00390876]),\n",
       " 'std_train_score': array([0.00100345, 0.00086851, 0.00057413, 0.00101623, 0.00073763,\n",
       "        0.00093597, 0.00104893, 0.0014662 , 0.00042381, 0.00087715,\n",
       "        0.00100779, 0.00088372, 0.00160667, 0.0012634 , 0.00128179,\n",
       "        0.00061829, 0.00089658, 0.00052264, 0.00100071, 0.00076123,\n",
       "        0.00095863, 0.00057904, 0.00056281, 0.00101256])}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gsearch3_1.cv_results_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split4_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best: -0.581140 using {'colsample_bytree': 0.9, 'subsample': 0.8}\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEKCAYAAAA4t9PUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xd8zdf/wPHXySYkIZEIMRJUkWWvr72JkdAoRWJrq2hLrSrVpa0falXVrC3Urr2K2kSQaBKJESJCEAmZ9/z+uJEGmZKbBOf5eNxHc+/9nPM550rvO59zPud9hJQSRVEURXlVegXdAEVRFOX1pgKJoiiKkisqkCiKoii5ogKJoiiKkisqkCiKoii5ogKJoiiKkisqkCiKoii5ogKJoiiKkisqkCiKoii5YlDQDcgPVlZWsmLFigXdDEVRlNfK2bNn70kpS2V13FsRSCpWrMiZM2cKuhmKoiivFSHE9ewcp4a2FEVRlFxRgURRFEXJFRVIFEVRlFx5K+ZIFKUwSExMJCwsjLi4uIJuiqI8x8TEBDs7OwwNDV+pvAokipJPwsLCKF68OBUrVkQIUdDNURQApJTcv3+fsLAw7O3tX6kONbSlKPkkLi4OS0tLFUSUQkUIgaWlZa6ulFUgUZR8pIKIUhjl9vdSBZLMnF0OgXsKuhWKoiiFmgokGUlOhNOLYONAuBdU0K1RFEUptFQgyYi+Iby/GvSNYE0viHtU0C1SlFx5+PAh8+fPf6Wys2bN4smTJ3nWlmXLljF8+PA8qy+7Dh06hJubW47L5bS9165dY/Xq1Tk+T3ZFRUXRpk0bqlSpQps2bXjw4EG6x924cYO2bdtSrVo1qlevzrVr13TSHhVIMmNRDjz/gAehsHEwaJILukWK8soKUyB502UWSJKSknJd/7Rp02jVqhVBQUG0atWKadOmpXtcv379GDNmDAEBAZw6dQpra+tcnzs96vbfrFRsDB1+hB2fw4FvofXkgm6R8gb4ettl/G9H52md1cuYMblzjQzfHzduHFevXsXV1ZU2bdpgbW3N+vXriY+Px93dna+//prY2Fg8PT0JCwsjOTmZSZMmERERwe3bt2nRogVWVlYcPHgw3fp37drFhAkTSE5OxsrKiv379xMVFcWAAQMICQmhaNGiLFy4EGdn5+fK+fj48PXXX6Ovr4+5uTl///03165do2/fvsTGxgIwd+5cGjVqxKFDh5g8eTI2Njb4+vri4eGBk5MTv/zyC0+fPmXz5s1UqlQJb29vTExMuHz5MhEREcyYMeOlK5HY2Fg++eQTLl68SFJSElOmTKFr164Zfn43b96kffv2hIaG0rt3byZPnsykSZOwsrJi5MiRAEycOBEbGxtWr15NQEAArq6ueHl5UaJECXbs2EFcXByxsbEcOHCAn3/++aXPH2DlypXMnj2bhIQE6tevz/z589HX13+uLVu2bOHQoUMAeHl50bx5c3788cfnjvH39ycpKYk2bdoAUKxYsQz7llsqkGRHnYFw5yIcnQGlHcGxe0G3SFFybNq0aVy6dAlfX1/27NnDhg0bOHXqFFJKunTpwt9//01kZCRlypRhx44dADx69Ahzc3NmzJjBwYMHsbKySrfuyMhIBg8ezN9//429vT1RUVEATJ48mZo1a7J582YOHDhAv3798PX1fa7s1KlT2b17N2XLluXhw4cAWFtbs3fvXkxMTAgKCqJXr16piVcvXLhAQEAAJUuWxMHBgUGDBnHq1Cl++eUX5syZw6xZswDtVcHhw4e5evUqLVq0IDg4+Lnzfvfdd7Rs2ZIlS5bw8OFD6tWrR+vWrTE1NU23j6dOneLSpUsULVqUunXr0qlTJwYOHIiHhwcjR45Eo9Gwdu1aTp06hbOzM9OnT2f79u2Admjs+PHj+Pn5UbJkSfbs2UNQUNBLn3+pUqVYt24dx44dw9DQkI8++ohVq1bRr18/Bg0axLBhw6hTpw4RERHY2toCYGtry927d19qb2BgIBYWFnh4eBAaGkrr1q2ZNm3aS0EpL6hAkh1CQIef4e4V2PwxWFYBW+esyylKBjK7csgPe/bsYc+ePdSsWROAmJgYgoKCaNKkCaNHj2bs2LG4ubnRpEmTbNV34sQJmjZtmrqgrWTJkgAcPXqUjRs3AtCyZUvu37/Po0fPzzc2btwYb29vPD098fDwALRZAIYPH46vry/6+voEBgamHl+3bt3UL9FKlSrRtm1bAJycnJ67WvL09ERPT48qVarg4ODAlStXXvoMtm7dyvTp0wHtOp8bN25QrVq1dPvYpk0bLC0tAfDw8ODo0aOMGjUKS0tLzp8/T0REBDVr1kw9Jr3yzz6XjD5/Pz8/zp49S926dQF4+vRp6nDUokWL0q03I0lJSRw5coTz589Tvnx5evbsybJlyxg4cGCO6skOFUgysfn8LWzMTGhYyRIMjKDnCljYHNZ+AEMOgmn6f50pSmEnpWT8+PEMHTr0pffOnj3LX3/9xfjx42nbti1fffVVtupLby2ClPKl1148bsGCBZw8eZIdO3bg6uqKr68vc+bMwcbGhgsXLqDRaDAxMUk93tjYOPVnPT291Od6enrPzT+8eJ4Xn0sp2bhxI1WrVs2yf5nVN2jQIJYtW8adO3cYMGBAhuXTXulk9PnPmTMHLy8vfvjhh0zbYmNjQ3h4OLa2toSHh6c792FnZ0fNmjVxcHAAoFu3bpw4cUIngURNtmcgIUnD/EPBDP7jDJdupfwFVcwaeq6E2Luw3kt7i7CivCaKFy/O48ePAWjXrh1LliwhJiYGgFu3bnH37l1u375N0aJF6dOnD6NHj+bcuXMvlU1Pw4YNOXz4MKGhoQCpQ1tNmzZl1apVgPaOKSsrK8zMzJ4re/XqVerXr8/UqVOxsrLi5s2bPHr0CFtbW/T09FixYgXJyTm/0cXHxweNRsPVq1cJCQl5KWC0a9eOOXPmpAa78+fPZ1rf3r17iYqKSp2Lady4MQDu7u7s2rWL06dP065dOyDrzyujz79Vq1Zs2LAhdagqKiqK69df3hKkS5cuLF++HIDly5enO7dTt25dHjx4QGRkJAAHDhygevXqmfbxVek0kAgh2gsh/hVCBAshxqXzvrcQIlII4ZvyGJTmvZ+EEJeFEAFCiNlCq3iaY32FEPeEELN00XYjAz2WD6iHeRFDvJacIvSedtKPsrWg82y4fhR2T9DFqRVFJywtLWncuDGOjo7s3buX3r1707BhQ5ycnOjRowePHz/m4sWL1KtXD1dXV7777ju+/PJLAIYMGUKHDh1o0aJFunWXKlWKhQsX4uHhgYuLCz179gRgypQpnDlzBmdnZ8aNG5f65ZfWmDFjcHJywtHRkaZNm+Li4sJHH33E8uXLadCgAYGBgRnOW2SmatWqNGvWjA4dOrBgwYLnrmoAJk2aRGJiIs7Ozjg6OjJp0qRM6/vf//5H3759cXV1pXv37tSpUwcAIyMjWrRogaenZ+r8g7OzMwYGBri4uDBz5syX6mrbtm26n3/16tX59ttvadu2Lc7OzrRp04bw8HBAe+XzbJ5o3Lhx7N27lypVqrB3717GjdN+vZ45c4ZBg7Rfo/r6+kyfPp1WrVrh5OSElJLBgwfn+HPMFimlTh6APnAVcACMgAtA9ReO8QbmplO2EXAspQ594DjQPJ3jzgJNs2pL7dq15asKvvtY1py6Rzaetl/eefT0vzd2TZByspmUZ5e/ct3K28Xf37+gm/DW8PLykj4+PvlyruTkZOni4iIDAwPz5Xy6kt7vJ3BGZuP7XpdXJPWAYClliJQyAVgLZHxv3fMkYII2ABkDhkBE2gOEEFUAa+BInrU4HZVKFWNZ/7o8iE2g3+JTPHqSMpzV+muo1BK2fwY3TuqyCYqiFFL+/v5UrlyZVq1aUaVKlYJuToHR5WR7WeBmmudhQP10jusuhGgKBAKfSilvSimPCyEOAuGAQHvVEvBCuV7AupSoqVPOdhYs7FeH/ktPM2D5aVYMrEdRIwPosQQWtoD1fWHIITAro+umKEqBq1+/PvHx8c+9tmLFCpycnAqoRS9btmzZK5XbvXs3Y8eOfe41e3t7Nm3alO7x1atXJyQk5JXO9SbRZSBJL53ki1/624A1Usp4IcQwYDnQUghRGagG2KUct1cI0VRK+Xeasu8DfTM8uRBDgCEA5cuXf8Uu/KdxZSt+ed+Vj1ef46NV5/i9Xx0Mi5SAXmtgUWvtnVz9d4KhSdaVKcpr7OTJN/cKvF27dqkT5kr26XJoKwwol+a5HXA77QFSyvtSymd/2vwO1E752R04IaWMkVLGADuBBs/KCSFcAAMp5dmMTi6lXCilrCOlrFOqVKnc9wbo4GTLd+5OHPo3ktE+F9BoJFhXA4+FcPscbBsJur9AUhRFKVR0GUhOA1WEEPZCCCO0VxBb0x4ghLBN87QL8Gz46gbQTAhhIIQwBJqleQ+0w1prdNbyTPSqV54x7aqyxfc2U7f7ayf93+0EzSeA31o48Wq5jBRFUV5XOhvaklImCSGGA7vR3nm1REp5WQgxFe2dAFuBEUKILkASEIX2Li6ADUBL4CLa4bBdUsptaar3BDrqqu1Z+ah5JaJiE1h8NBRLUyM+aVUFmo6BO36w50uwrg6V0r9NUlEU5U2j05XtUsq/gL9eeO2rND+PB8anUy4ZeHnJ7X/vO+RhM3NMCMHEjtV4EJvA/+0NxMLUiL4NKoD7AljcFny8tSvfSxZoMxVFUfKFWtn+ivT0BD/2cKblu9Z8teUS2/1ug3FxeF+7ipc1vSE+45WtipLfClMaebUfSe5kZz+SgwcP4urqmvowMTFh8+bNOmmPCiS5YKivx7zetahToQSfrvPlSFCk9irkvWVw71/YNAw0moJupqIAhSuQvOkKw34kLVq0wNfXF19fXw4cOEDRokVTE1zmNZW0MZeKGOmzyKsuPX87ztAVZ1k9uAGulVpA22+1KVT+/hmaj826IuXtsnOcdmuCvFTaCTqkv8ERqP1I3rb9SNLasGEDHTp0oGjRohkekxsqkOQB8yKG/DGgHt0X/EP/pafwGdaQyg0+0n5RHPoebGpAtZxfTitKXlL7kbxd+5GktXbtWj777LNMj8kNFUjyiLWZCSsH1qf7r8fpu/gUGz5sRFm3WRD5L2waCpb7tGtOFAUyvXLID2o/kjd/P5JnwsPDuXjxok4XWqpAkocqWJqyfEBd3v/tBH0Xn2TDsEaUfH+Vdg+TNb1g8AEoWrKgm6koaj8S3vz9SJ5Zv3497u7uGBoaZlpnbqjJ9jxWo4w5i7zqcOvBU/ovPUWMccoeJtG3YMMASM79RJuivAq1H8nbtR/JM2vWrKFXr16Z9i23VCDRgfoOlsztXYtLt6MZtuIs8ba1odP/QchB2De5oJunvKXUfiRv134koJ0nunnzJs2aNcvx55cTIh+S5xa4OnXqyGf/APlpw9kwRvtcoKNTaeb0qoX+ri/g1EJwXwguPfO9PUrBCggIyHD8Xclb3t7euLm50aNHD52fS6PRUKtWLXx8fF7rVPLp/X4KIc5KKetkVVZdkehQj9p2TOxYjb8u3mHSlkvItt9BxSaw9RO4da6gm6coSi6p/Ui01GS7jg1u6sD92AQWHL6KpakRn7+3TLuHybo+MPggFLcp6CYqSo6o/Uj+o/Yj0VKBJB+MbV+VB7EJzDkQTElTI/q/v0qbk2t9P/DaBgZGBd1ERck2tR+J8iI1tJUPhBB85+5Iuxo2fL3Nn813LKHbfLh5Av4arfYwURTltaYCST4x0Nfjl/dr0tDBktE+Fzho8D/432dwbjmcWVzQzVMURXllKpBk4mnS0zytz8RQn4X9avOubXE+XHWWMw4fQZV2sHMsXDuWp+dSFEXJLyqQZEBKyUf7PmLkgZHcjL6ZZ/UWNzFkWf962JoXYcAf5wj83wwoYa+dL3l4I8/OoygvKkzZf1Ua+dzJThp5gC+++IIaNWpQrVo1RowYkW6mgbygAkkGkmUyjcs25nj4cbpu6cqMszOISYjJk7qtihnzx4B6FDHSp8/KK4R3XALJCbD2A0hQqboV3ShMgeRNVxjSyP/zzz8cO3YMPz8/Ll26xOnTpzl8+HCuz50eFUgyYKBnwCCnQWx3304H+w4svbQUt01ubArahEbmfo+RciWLsmJgfeKTNLz/530edVygzRa8dbiafFd0Im0a+TFjxvDzzz9Tt25dnJ2dmTxZm3EhNjaWTp064eLigqOjI+vWrWP27NmpaeQzWtkO2jTytWrVwsXFhVatWgHav5y7deuGs7MzDRo0wM/P76VyPj4+ODo64uLiQtOmTQHtF3GTJk2oVasWtWrV4p9//gG0VxTNmjXD09OTd955h3HjxrFq1Srq1auHk5MTV69eBbQLEocNG0aTJk145513UrPwphUbG8uAAQOoW7cuNWvWZMuWLZl+fs/SyFetWjU15fukSZP45ZdfUo+ZOHEis2fPZty4cRw5cgRXV1dmzpzJsmXLeO+99+jcuXNqksn0Pn/QppF/ll1g6NCh6aaH2bJlC15eXoA2jXx6G1YJIYiLiyMhIYH4+HgSExOxsdHNcgN1+28WrIta893/vqPXu72YdmoaX/3zFWuurGFcvXHUsqmVq7rfsSnOEu+69Fl0kl6HzPiz2ZeYHP5Gu6/E/z7Nox4ohdGPp37kStSVrA/MgXdLvsvYehnvfaPSyL9daeQbNmxIixYtsLW1RUrJ8OHDdZZZQV2RZJOjlSMrOqxgWpNpRMVF4bXLizGHxxAeE56remtXKMGvfWoRGPEYr38bkVzdHfZ9DYF78qjlivKytGnMa9WqxZUrVwgKCsLJyYl9+/YxduxYjhw5grm5ebbqyyyNfN++fYGs08j//vvvqX99JyYmMnjwYJycnHjvvffw9/dPPf5ZGnljY+OX0shfu3Yt9bjspJGfNm0arq6uNG/ePDWNfEaepZEvUqRIahr5ihUrpqaRf/Z55jSNfNrPf//+/alp5F1dXdm/f3/qgsdFixal5vfKjuDgYAICAggLC+PWrVscOHCAv//+O9vlc0JdkeSAEIJODp1oUa4Fyy4vY+mlpRy8eRDvGt4McBxAUcNX232seVVr/s/ThVHrfBlhMpC5NsGIjYO0aeetKudxL5TCILMrh/yg0si/+WnkN23aRIMGDShWrBgAHTp0SA34eU1dkbyCooZF+cj1I7Z220rL8i35ze83Om/uzLar2155/qSra1mmdK7BjivRfG82CalvAGt7QdyjrAsrSjaoNPJvVxr58uXLc/jwYZKSkkhMTOTw4cNqaKsgJFy7hiYuLsP3bYvZ8lPTn/ijwx9YFbFiwtEJ9N3Zl4uRr7YXt1ejioxoVYXfLyaxqvw3EBUCfw4BTe4n9xVFpZF/u9LI9+jRg0qVKuHk5ISLiwsuLi507tw5x59jtkgpdfYA2gP/AsHAuHTe9wYiAd+Ux6A07/0EXAYCgNn8l/LeCFgIBAJXgO5ZtaN27doypzQJCTK4bTsZ3K69jD17NsvjkzXJclPQJtl8XXPpuMxRjv97vLwTcyfn59Vo5JebLsoKY7fLv1d9L+VkMyn3Tc1xPUrh4+/vX9BNeGt4eXlJHx+ffDlXcnKydHFxkYGBgflyPl1J7/cTOCOz8V2vsysSIYQ+MA/oAFQHegkhqqdz6DoppWvKY1FK2UZAY8AZcATqAs92ZpkI3JVSvpNSr05ujBaGhth8NQmZkMD1D/pw57vv0cTGZni8ntCjW+VubHffziCnQey6tovOmzuz0G8hcUkZX9W8dF4hmNKlBm7OtvT1cyKkXHc4Mh0up599VFGUgqPSyGvpcrK9HhAspQwBEEKsBboC/pmW0pKACdqrDwEYAhEp7w0A3gWQUmqAe3nb7P8Ua9wYh21buTtzFg9WrCDmwAFsv/0G04YNMyxjamjKyFoj8ajiwYwzM5hzfg4bAzfyWZ3PaFuhbboTki/S1xPM8HTl0dNEOgZ34XiZEEps/ggsK2tvDVaUAqTSyP9HpZHX0tkOiUKIHkB7KeWglOd9gfpSyuFpjvEGfkA7vBUIfCqlvJny3nRgENpAMldKOVEIYQFcBHyA5sBVYLiU8lmQSXv+IcAQgPLly9dOb8IqJ56cOUP4xC9JuH4di/d6YP3FF+gXL55luVPhp/jx9I8EPgiktk1txtYdSzXL7E14xcYn8cGik9wNv8FBsykYGxrBkENgmv7thUrhpnZIVAqzwrpDYnp/er8YtbYBFaWUzsA+YDmAEKIyUA2wA8oCLYUQTdFeQdkBx6SUtYDjwPT0Ti6lXCilrCOlrFOqVKlcd6ZonTrYb9mM5aCBPNz4JyFunXl88GCW5erZ1mO923omNZhEyMMQem7vyeR/JnPvadYXUqbGBiz1rkvRkmXwih2BJiYCfLwgOTHX/VEURckrugwkYUC5NM/tgNtpD5BS3pdSPrtG/h2onfKzO3BCShkjpYwBdgINgPvAE+DZdaYPkLvl5TmgZ2KC9ejRVFy3Fn0zM8I+/IhbY74gKYOEac/o6+njWdWT7R7b6Vu9L1uDt+K2yY2ll5aSkJyQadkSpkb8MaAeN0zeZYocAteOwO6JedktRVGUXNFlIDkNVBFC2AshjID3ga1pDxBC2KZ52gXtHVoAN4BmQggDIYQh2on2gJS7CLahHdYCaEX25lzyVBEnJ+w3bsDq44+J3rmTELfORO/anWU5MyMzxtQdw59d/6S2TW1mnJ1Bty3dOHDjQKZZOctYFOGPgfXZLpqx1qALnPoNzq3Iyy4piqK8Mp0FEillEjAc2I02QKyXUl4WQkwVQnRJOWyEEOKyEOICMALt7cAAG9DOf1wELgAXpJTbUt4bC0wRQvgBfYHPddWHzAgjI0p9Mhz7jRswLF2aW6NGEfbJCJIiI7Msa29uz7xW81jQegGGeoaMPDiSIXuHEPQgKMMyla2Lsax/Xb5PeJ+zBq7IHZ/BzVN52SVFUZRXk517hF/3x6usI8kJTWKijFy4UAY4Ocsr9erLB5s2SY1Gk62yCckJcqX/StlodSPpvNxZfnP8G/ng6YMMjz8aFCnrTFgvw6e+IzU/VZHy0a286oaiYwW9juTBgwdy3rx5r1R25syZMjY2Ns/asnTpUvnxxx/nWX3ZdfDgQdmpU6ccl8tpe0NDQ+WqVatyfJ7sun//vmzdurWsXLmybN26tYyKikr3uC+++ELWqFFD1qhRQ65duzbTOgvlOpK3iTAwwGrwYOw3b8bYwYHwceO5OXQoieFZJ3Q01DPkg2ofsMN9B57veLIhcAOdNnViVcAqEjUvT6o3rmzF1+//D6+nnxL/5BGatX0gMfvrVJS3l9qPJP8Uhv1IduzYwblz5/D19eXkyZP8/PPPREdH5/rc6VGBJA8ZO9hTYeUKbCZM4MnpM4S4debBuvXZ2pXMwsSCiQ0msqHzBmpY1mDaqWn02NqDo7eOvnRsRydbvLp2ZGT8h+jdPovcPkrtYaJkSe1H8rw3fT8Sf39/mjVrhoGBAaampri4uLBr165M+/jKsnPZ8ro/dD20lZ74GzfkNS9v6V/1XXmtn5eMv3Ej22U1Go08cP2A7Lixo3Rc5ig/3PuhDHkY8tJxcw8EyZkTvaWcbCY1/7zakIWSf9IOHYR/95281qdvnj7Cv/su0/OHhobKGjVqSCml3L17txw8eLDUaDQyOTlZdurUSR4+fFhu2LBBDho0KLXMw4cPpZRSVqhQQUZGRmZY9927d6WdnZ0MCdH+nt6/f19KKeXw4cPllClTpJRS7t+/X7q4uEgpnx8qcnR0lGFhYVJK7fCblFLGxsbKp0+fSimlDAwMlM/+Hz548KA0NzeXt2/flnFxcbJMmTLyq6++klJKOWvWLDly5EgppTZFSrt27WRycrIMDAyUZcuWlU+fPn1uaGv8+PFyxYoVqeetUqWKjImJSbd/S5culaVLl5b37t2TT548kTVq1JCnT5+WoaGhsmbNmlJKbaoUBwcHee/evZeG0JYuXSrLli2b+rlk9Pn7+/tLNzc3mZCQIKWU8sMPP5TLly+XUko5cOBAefr0aSmllObm5s+1z8LC4qU27969WzZq1EjGxsbKyMhIaW9vL6dPn57RP2GuhrZUGnkdMSpXjvJLl/DQx4e7P/5ESJeuWH86ihIffIBISeyWESEELcq3oHHZxqwOWM1vfr/hscWDXtV6McxlGGZG2uypHzWvxDePP2f36Ru02f0lwqY6ODTXfeeU117a/TAAYmJiCAoKokmTJowePZqxY8fi5uZGkyZNslVfZvuRbNy4Ech6PxJPT088PDwA7X4kw4cPx9fXF319fQIDA1OPf7YfCfDSfiQH06ztys5+JFu3bmX6dO1StGf7kWS0aPTZfiRA6n4ko0aNSt2PJCIi4pX2I4H/Pn8/P7/U/UgAnj59mpoiftGiRenWm5G2bdty+vRpGjVqRKlSpWjYsCEGBrr5yleBRIeEEJTw9KRY06aET55MxPc/EP3XTmy//w5jB4csyxvpG+Ht6I1bJTfmnp/LSv+VbL+6neE1h9O9Snf09fT50q0GE2K+oeKVwVRc0w/jDw9DSft86J2SG6UnTCjQ80up9iOR8s3ejwS0Q20TJ2rXnfXu3Vtn+cDUHEk+MCxdmnILFlDmpx9JCA0ltJs7935biMzmpJtVESumNJrCOrd1OFg48M2Jb3hv+3ucDD+Jnp7gm54NWFLue54mJBG97D2Ij9Fxj5TXkdqP5O3ajyQ5OZn79+8D4Ofnh5+fX+rVW15TVyT5RAiBeZcumDZqxJ1vviVy5kwe796N7fffYfLuu9mqo5plNZa2W8re63uZcXYGg/YMolX5Vnxe53OmeLnx8/ybTHzwJfdWDsCq/1rQU38nKP9Jux9Jhw4dUvfDAChWrBgrV64kODiYMWPGoKenh6GhIb/++ivw334ktra2zw0fPZN2PxKNRpO65/qUKVPo378/zs7OFC1aNMP9SIKCgpBS0qpVq9T9SLp3746Pjw8tWrTI1X4kERERGe5HMmrUKJydnZFSUrFixXQn5Z95th9JcHAwvXv3fmk/EgsLi3T3I/H29qZEiRLP1dW2bVsCAgJe+vzT7kei0WgwNDRk3rx5VKjawPzpAAAgAElEQVRQ4bk928eNG4enpyeLFy+mfPny+Pj4ANr9SBYsWMCiRYtITExMHZo0MzNj5cqVOhva0lnSxsKkTp068tmGMIVF9O493Jk6leRHj7AaMhjLYcPQMzLKdvn45Hj+uPwHv1/8nSRNEv2q96NnFW+2zZ/KkLjFhNf6DNsuk7OuSMk3Kmlj/vH29sbNzY0ePXro/FwajYZatWrh4+PzWqeSL6xJG5VMmLVri8P2bZh36si9+b9yrXt3nqZza2RGjPWNGew8mO3u2+lg34HFlxbTe5c7Jh0b8Jd+M2zPzSD85AYd9kBRFLUfiZa6IikEYg4fJnzyFJLu3qWktzelRnyC3guX4Vm5GHmRaaen4RfpR2Wzd+hz5Srt48OJ7bsb60quOmq5khNvyhXJ67AfyavK6X4kb5LcXJGoQFJIJD9+zN2fp/Nw/XqMKlTA9rtvKVony3+/50gp2RG6g5lnZ3L3yV1axCTg/UBg/+HflLBM/64OJf+8KYFEeTOpoa03gH7x4thO/Zryy5Yik5O53qcvd6Z+Q3JMxtv7vkgIgZuDG9u6bWOYyzCOFi/KkLIGzFjtRuTjhzpsvZJdb8MfbsrrJ7e/lyqQFDKmDRrgsHULJfr15cGaNYR26ULMsWM5qqOoYVE+dv2Y7R47qGVSic1mT+nm05LNQVvVF1kBMjEx4f79++rfQClUpJTcv3//pbvackINbRViT86dJ3ziRBJCQzHv7oHN2LHov3APfnZsXdqLlYnnCDA2wrmUC+PrjcPRylEHLVYyk5iYSFhYGHFxKsmmUriYmJhgZ2eHoaHhc6+rOZI0XtdAAqCJj+fe3HncX7IEA0tLSk+ZTPGWLXNWSXIit+a04VjSVaaXKstTYulSqQsja43EuqiaO1EUJX1qjuQNoWdsjPXnn1Fx3Tr0S5Qg7KOPufX56Cy3932OviFlB2+gXUJRVoVGUNOkHTtDd+K2yY2FfguJS1J/ISuK8upUIHlNFHGsgb3Peqw+GU70nj2EdHIjeufO7I+3m1ph5r2OCiKWTy4fxqvsLBqVacSc83Pourkre67tUWP3iqK8EhVIXiPCyIhSH3+s3d63bFluffoZYZ98QmJKXp4sy5dxRd99PnX1ArE6MIvmFmNY1HYRpkamfH74c/rv7s+VqCtZV6QoipKGCiSvIZN33qHimtVYjxlN7JGjhLh15uGfm7J1RaHv3IOkRqP4wGA/ZzdO50m0Pevd1jOpwSSuPryK5zZPvj3xLY8TMk44pyiKkpYKJK8pYWCA5cCB2G/ehHGVKoRPmMDNwUNIvH07y7IGrb8iyaE1kw2Ws2jlSi7cjMazqic7PHbQu1pvfAJ96Lq5K/uu71PDXYqiZEkFktecsb09FVb8gc2XX/Lk3Dnt9r5r1iA1mowL6elj8N5iKFGRuQazGL90J1fuRGNmZMa4euNY1XEVJU1K8umhTxl5cCR3Yu/kX4cURXntqEDyBhB6epTs8wEOW7dSxNWFO19P5YaXNwnp7GOQqogFBr3XYm6oYbaYzuBFR7gZ9QQARytH1rit4bPan3H89nG6bu7KqoBVJGtyvieEoihvPhVI3iBGdmUpt3gxtt9+Q9yVK4R07cb9pcuQGW0KVOod9LovoqoMZXzSPPouOkHkY20yPkM9Q/o79mdT103UtK7JtFPT6LuzL/9G/ZuPPVIU5XWg00AihGgvhPhXCBEshBiXzvveQohIIYRvymNQmvd+EkJcFkIECCFmi5R9LYUQh1LqfFZGrahLQwiBRY8eOGzfhmnDhtz98Ueu9e5NfHBw+gWqtke0/JKOHKNTzAa8l54iOi4x9W274nb82vpXpjWZxq2YW/Tc3pOZZ2fyNOlpPvVIUZTCTmeBRAihD8wDOgDVgV5CiOrpHLpOSuma8liUUrYR0BhwBhyBukCzNGU+SFMme/e+vmUMbWywmz+PMtOnk3j9BqHuHtxbsACZmPjywU0+h+rdGK23GpuII/RdfIoHsQmpbwsh6OTQiS1dt9ClUheWXFqCxxYP/rn9Tz72SFGUwkqXVyT1gGApZYiUMgFYC7y8sXD6JGACGAHGgCEQoZNWvsGEEJi7dcJhx3aKtW5F5KxfCPXsSVxAwIsHQrf5CBtHfis6n4TwADx/O86dR8+veLcwsWBq46ksbrsYfT19hu4dyoQjE4iKi8rHXimKUtjoMpCUBW6meR6W8tqLugsh/IQQG4QQ5QCklMeBg0B4ymO3lDLtt9/SlGGtSc+GvF4khBgihDgjhDgTGRmZJx16XRlYWmI3cyZl58wm6V4koe95cnfWLDQJ/111YGQK76/C0NCY7SZfUf/hX/T49RjX7r2cxr6ebT02dtnIUOeh7Ly2k66bu7IleIu6VVhR3lK6DCTpfcG/+E2zDagopXQG9gHLAYQQlYFqgB3a4NNSCNE0pcwHUkonoEnKo296J5dSLpRS1pFS1ilVqlSuO/MmMGvThkrbt2PeuTP3F/xGqIcHTy9c+O+AEhVg6GH07Wrxrd4Cvo77kYG/7iYgPPqluoz1jRleczg+bj5UNKvIl8e+ZPCewVyPzuROMUVR3ki6DCRhQLk0z+2A51bLSSnvSymf7dn5O1A75Wd34ISUMkZKGQPsBBqklLmV8t/HwGq0Q2hKNumbm1Pmh+8p9/tCNLFPuNarNxHTfkTzNGXy3NwO+m2FNt/QUu8ca5M/45fffuXs9fSHryqXqMzyDsuZ1GASl+9fxmOLB7/7/U5icjpzMYqivJF0GUhOA1WEEPZCCCPgfWBr2gOEELZpnnYBng1f3QCaCSEMhBCGaCfaA1KeW6WUNQTcgEs67MMbq1iTJjhs24pFT0+ili0jpGs3Yk+d0r6ppweNRyCGHMTC0poFfM/lxR9yxP9munXpCT08q3qypdsWmpVrxuzzs/Hc7onvXd987JGiKAUly0AihKgkhDBO+bm5EGKEEMIiq3JSyiRgOLAbbYBYL6W8LISYKoToknLYiJRbfC8AIwDvlNc3AFeBi8AF4IKUchvaiffdQgg/wBe4hfZKRnkF+sWKYTt5MuWXLwfgRj8vwr/+muSYGO0BpZ0wHHaYJ7WG0E9vF6XXtePvv/dnWJ91UWtmNJ/B7BazeZzwmH47+6m8XYryFshyYyshhC9QB6iINihsBapKKTvqvHV55HXe2Cq/aJ4+JfKX2UQtX46+lSXWn36GebeuCD3t3xqx/nuI3zCUYsmPuFxtJDU9vwQ9/Qzri02MZe75uawKWEWpIqWYUH8CrSq0yq/uKIqSB/JyYytNytWFOzBLSvkpYJtFGeU1o1ekCDbjxlJx/TqMypQlfMIErnn25Mn58wCYVm9LkRGn8DNtSM0rM7g1uy08TH+oC8DU0JSx9cayutNqSpiUYNShUYw4MELl7VKUN1B2AkmiEKIX4AVsT3nNMJPjlddYEScnKqxZTZmffiTp7l2u9+rNrS++IDEigiIWpXD+dCsrS4/F/MEl4uY0QF7ckGl9L+bt6ralG6sDVqu8XYryBsnO0FZ1YBhwXEq5RghhD/SUUk7LjwbmBTW09Wo0sbHc+/13opYsBX19rIYOoWT//khDI2au300L/y+prReExskTvY4/Q5HMp85uPr7Jtye+5Z/b/+Bs5cxXDb+iasmq+dQbRVFyKrtDW1kGkhcqLQGUk1L65aZx+U0FktxJuHmTuz/9zOO9ezEsWxbrsV9QrHVrZuzxRxyZySjDP9EzK4Pw+A0q/i/TuqSU/BX6Fz+d/ono+Gi8angxzGUYJgYm+dQbRVGyK8/mSFKSJJoJIUqivYNqqRBiRl40Unk9GJUrh92c2ZRfthS9okW5NWIkN/sPYLi9IcXbTcAjfgoRTzTIZW6wdzIkJWRYV9q8XW6V3Fh8aTEeWz04fvt4PvZIUZS8lJ05EnMpZTTgASyVUtYGWuu2WUphZNqgAfab/sTmq0nEX7lCqLsHXQ6tom+bVrSO/Za9Ju3h2CxY1BLuZr73u4WJBd80/obFbRejJ/QYsneIytulKK+p7AQSg5SFg578N9muvKWEgQEle/em0u5dlOjdmwfr1uM0fhALStxh5KM+TDH9Es2j27CwGZxcCFkMnT7L2zXEeUhq3q6tV7eqvF2K8hrJTiCZinb9yFUp5WkhhAMQpNtmKYWdvoUFpb+ciMPmTRSpUR3LJXNZf2YeV4KN6M50nto1hp1jYFUPeJz5Lb/G+sZ8UvOT1LxdE49OZPDewdyIvpFPvVEUJTdyNNn+ulKT7bolpSTmwAEipv1I4s2bnC7rxPq63Vjc5gE2x78BwyLQZTZU65xlXRqpYUPgBmaenUmiJpFhLsPwquGFoZ6641xR8lteTrbbCSE2CSHuCiEihBAbhRB2edNM5U0ghKB4q1Y47NhOqc8+o+79YL7f+gNL5wXg32Y9WJSDdX1gy3CIj8m0rrR5u5raNeWXc7/guc2TC5EXMi2nKErByc46kr1os+yuSHmpD9pU7m103LY8o65I8ldixF2uTfuZpJ3beWBSHKOhH1K78g3EsZlQoiJ4/A7l6marroM3DvLdye+4++QuPav2ZGStkRQzKqbbDiiKAuRtipRSUsqlUsqklMcyQG3woWTI0MaaKjN/xmzpCh6ZWWH6y09c+MWXpw3mgCYZlrSDgz9AclKWdbUo34It3bbwQbUPWPfvOrpu6cr+6xknjlQUJf9lJ5DcE0L0EULopzz6APd13TDl9Ve2YR0a/PUn69sMJCbsNtdGfc/tW21IrNAFDk/TBpT7V7Os51nerlUdV1HCWJu3a+SBkSpvl6IUEtkZ2ioPzAUaot3h8B9ghJTytbmlRg1tFawnCUkMX3yMcjt98Aw5jL6REVZdG1FSfwt6Igk6TIOafbV7x2chUZPICv8V/Or7K/p6+oysNRLPdzzRzyQTsaIor0YnKVLSVD5KSjnrlVpWAFQgKXjxScl8us6Xcycu80PEQUr7ncCwTGmsG0Bxw3OIam7QeTaYWmarvhfzdk1uNJl3Sryj414oyttF14HkhpSy/Cu1rACoQFI4JGskE/68yLozNxlTKpr2h9aQEBRE0WplsbG/hEkZM+g6H6pkL3GClJIdoTv4+fTPRMdH4+3ozVDnoSpvl6LkEV0HkptSynJZH1k4qEBSeEgpmbbzCr/9HUI3Jxsmaq4QNXcumsfRlHA0xKrSdQyaDIY2U7XrT7LhYdxD/u/s/7E5eDPlipdjUoNJNCzTUMc9UZQ3X17etZWeN38Vo6ITQgjGd6zGF+2rsvliBGOTq1J22w5K9OrNg8vJXN1dnqiVK5G/NoXw7K0dSS9v18SjE3kQ90DHvVEUBTK5IhFCPCb9gCGAIlJKA102LC+pK5LCadXJ63y5+RJ1K5RkkXcdjG6EEvH9Dzw5cQLjEhKbmo8w7f0FNBqR6ba+acUnx7PQbyFLLi6hmFExxtQdQ2eHzohsTOQrivI8nQ5tvW5UICm8tl24zafrfKlaujjLB9TD0tSImP37ifjhBxJv3aZY2afYdKqC0YDFYJH9abmgB0F8ffxrLkReoL5tfb5q8BXlzV6baT1FKRRUIElDBZLC7eC/d/lw5VnKmBdhxaD6lLUogiY+nqhly7g3fx4kJlCyRiKWo6egX79vtutVebsUJXdUIElDBZLC78y1KPovO00xYwNWDKxPZWttGpTEiLtETpvKo537MTBJplSnqphPWIbI5m3CAHef3GXaqWnsvb6XKiWqMLnhZFxKueiqK4ryxtD1ZHt2G9FeCPGvECJYCDEunfe9hRCRQgjflMegNO/9JIS4LIQIEELMFi8McgshtgohLumy/Ur+qVOxJOuGNCQxWYPnb8e5GPYI0KZbKTNzLhVXr8SgtA3hG4O51uF/PN25PNt1Wxe1ZkbzGcxuMZvo+Gj6/tWX709+T0xC5gkkFUXJHp0FEiGEPjAP6ABUB3oJIaqnc+g6KaVrymNRStlGQGPAGXAE6gLN0tTtAahvgTdM9TJm+AxrRBFDfXr9foITIf9l4ilSqzYV/zqM7dihJMbAtU+ncbtfexJv38x2/c/ydvWu1pu1V9Zq83bdUHm7FCW3spNG/rEQIvqFx82U1PIOmRStBwRLKUOklAnAWqBrNtslARPACDAGDIGIlPYUAz4Dvs1mXcprxN7KlA0fNqS0uQleS06xzz8i9T2hp4dF/1FU2n8Yy5b2RJ+5xtV2bbk341s08fHZqt/U0JRx9cb9l7fr4ChGHRxF0AO1V5uivKrsXJHMAMYAZQE7YDTwO9rAsCSTcmWBtH8uhqW89qLuQgg/IcQGIUQ5ACnlceAgEJ7y2C2lDEg5/hvg/4An2Wi78hqyNS/C+qENebd0cYauPMum82HPva9fwhrr+X/hMH8iprYaIheuIqR1Mx7v2ZvtLXqdSjmxxm0Nn9b+lGO3juGx1YNe23ux/t/1PE54rItuKcobKzuBpL2U8jcp5WMpZbSUciHQUUq5DiiRSbn0btx/8f/ybUBFKaUzsA9YDiCEqAxUQxu4ygIthRBNhRCuQGUp5aasGi2EGCKEOCOEOBMZGZllJ5XCpaSpEasGN6BexZJ8uu4Cy/+59tIxRs37Uu7PI5T/wB69hHuEjRjBjX59iAsMzNY5DPUMGeA4gD099jC27ljiNfF8c+IbWq5vyfgj4zl95zQaqcnjninKmyc72X+PAzOBDSkv9QA+k1I2EEL4SildMyjXEJgipWyX8nw8gJTyhwyO1weipJTmQogxgImU8puU974C4oDHwCQgATAArIF/pJTNM+uDumvr9RWXmMwna86z1z+Cz9q8wyctK7+8uFBK5MlFPJj3LZEXiqBJ0qdEr16U+mQ4+hYW2T6XlBL/+/5sCt7EXyF/8TjxMXbF7OhWuRtdK3eltGnpPO6dohRueXb7b8o8yC9o08gDHAc+BW4BtaWURzMoZwAEAq1Sjj0N9JZSXk5zjK2UMjzlZ3dgbEqA6gkMBtqjvbLZBcySUm5LU7YisF1K6ZhVJ1Ugeb0lJWsYu/EiG8+F0b9xRSZ1qo6eXjoXvPeCSFo5gHv7r/Mg2BR9MzOsRoygRM+eCIOcJWKIS4pj3419bA7azMk7J9ETejQs0xCPyh40L9ccI32jPOqdohRehWIdiRCiIzAL0AeWSCm/E0JMBc5IKbcKIX4AugBJQBTwoZTySsrVyXygKdrhsF1Sys9eqLsiKpC8NTQayTc7/Fl67Brda9nxY3cnDPTTGZlNSoDDPxK3fTYRF0rx5LbEuEoVbCZOwLRBg1c6983HN9kSvIXNwZuJeBKBhbEFbg5uuFdxV6nrlTdaXl6R2AFz0N6OK4GjwEgpZVimBQsRFUjeDFJK5hwIZsbeQNpUt2FOr5qYGGaQg+v6ceSfg3l8+R53A8qReD+G4m1aY/3FFxiVe7XE1cmaZE6En2BT8CYO3DhAoiaRGpY18KjiQXv79pgZmeWid4pS+ORlINkLrAZWpLzUB/hAStkm163MJyqQvFmW/3ONyVsv09DBkt+96lDMOINhq7hHsHMsmnNriAqvyr2ziZCsoWT//lgNGYyeqekrt+Fh3EN2hO7gz6A/CXwQiLG+Ma0rtMajsgd1StdBT+h0ra+i5Iu8DCQvTahnNsleGKlA8ubZfP4Wn/tcwLGMGUv716OkaSZzFpf+hO2fkvg4kciIRjw6ehmDUqWwHv05Zp07I/Re/UtfSol/lD+bgtQEvfLmyctAsg9YBqxJeakX0F9K2Sq3jcwvKpC8mfb5R/Dx6nOUK1mUFQPrYWueyUZYj27B5g8h9DBPijYj4qQBcZcDKOLigs3ECRRxds51e+KS4th/Yz+bgjdxMvwkAkGjMo1wr+JOi3It1AS98trJy0BSHpiL9q4tCfwDjJBS3siLhuYHFUjeXCdC7jNo+RnMixiyclB97K0yGa7SaODkr7BvCtLYgkdmXtxdtZvkyHtYDh5EqVGjEPrZ2/ckK2GPw9hyVTtBfyf2TuoEfbfK3ahasmqenENRdE3XW+2OklLOeqWWFQAVSN5sF8Me4bX0FHpC8MeAelQvk8Wkd8Rl2DgY7l4m2ak/d8+b8nDDn5g2bULZ6dPRN8u7SfNkTTInw0+yKXgT+2/sJ1GTSHXL6nhU9qCDQwc1Qa8UaroOJDeklK/NLkEqkLz5gu/G0G/xSR7HJ7HUuy51KpbMvEBiHOyfCifmgdU7PIhvwZ2lf2FkVw67+fMwdsgsjdyreTZBvyloE/8++Dd1gt69sjt1S9dVE/RKoaPrQHJTSvlq91AWABVI3g63Hj6l76KT3H70lF/71KZFVeusC109CNtGwMMbPHlgQdhRc6TUp8z06RRvqZtpQCklAVEB/Bn0Z+oEfdliZbUT9JW6YlvMVifnVZScUlckaahA8va4FxOP15JT/HvnMTN7utLZpUzWhTTJcP0Y+K0j8dQ2wg4YEvfAkFJujlh+/hXCNvcT8RnJaIK+W5VutCzXUk3QKwUq14FECPGYl5MsgjZlSREpZc5yThQgFUjeLtFxiQxadobT16P4tpsjH9SvkP3CiU/R+G0h/IdfiL70kOLlnlLGzRa92j3B6T0wTy+Bdd54cYLe3Nhcu4K+sruaoFcKRKFIkVJYqEDy9olLTOajVec4cOUuX7SvyofNKr2c7DETUkqiFszl7uz5GFsZYlc/DKNiGqj4P3DuCdW7gIm5Ttqe0QS9e2V3Oth3wNxYN+dVlBepQJKGCiRvp8RkDaN9LrDF9zZDmzowrsO7OQomADFHjnDrs88R+oKy/RtiGn8YokLAwASqdtAGlUqtwEA3Q1CP4h+xPWT7cxP0rcq3wr2KO/VK11MT9IpOqUCShgokby+NRjJl22X+OH6dnnXK8b2HE/rpZQ7ORHxoKGEfDyfh+nVsxo+jRLOqiIvr4dJGeHIfipQERw9tULGrCzkMVtnlf1+7gn5H6A4eJ2gn6LtW7kq3St3UBL2iEyqQpKECydtNSsnMvYHMPhBMR6fSzOzpirFBzhYeJsfEcHv0GGIOHcK8R3dKf/UVevoCrh4Av3VwZQckxUGJitqA4twTLCvppD/xyfHsv66doD8RfgKBoGGZhrhXdqdleTVBr+QdFUjSUIFEAVh0JIRvdwTQpIoVC/rUxjSjZI8ZkBoNkbNnc3/BbxRxdaXs7F8wtE65xTguGq5shwtrIfRvQELZOtqA4ugBplZ53yHgVsyt1BT34bHhmBub08m+E+5V3Hm35Ls6Oafy9lCBJA0VSJRn1p+5ybiNfriUs2Cpd10siub8r/foXbu4PX4C+mZm2M2dQxEnpxcOuA0XN4Dfeoi4CEIfKrcGZ0+o2hGMiuZRb/6jkRpOhJ9gc9Bm9t3YR6ImkWolq+FexZ2O9h3VBL3ySlQgSUMFEiWtXZfuMGLNeeytTFkxsB7WZiY5riPuyhXCPh5OUmQkpad+jUW3bukfGHFZG1Au+kD0LTAqrr3jy9kTKjYBvbzJ7ZXWo/hH7AjZwabgTVyJuoKRnhGtKrTCvbI79W3rqwl6JdtUIElDBRLlRceC7zH4jzNYFTNm5cD6lLfM+VVC0oMH3Br1KU9OnqSklxfWY0ZnvKWvRgPXj2rnU/y3Qnw0FLcFpx7a4S8bR51M0gfcD2BT8CZ2hOwgOiGaMqZlUlPclymWjcWayltNBZI0VCBR0uN78yHeS09hpK/H6LZVaVnNGqtixjmqQyYmEvHjTzxYuRLTRg0pO2MG+hYWmRdKfAqBu7RXKkF7QJME1tW1VylO74G5XS56lb745HgO3DjAn0F/cjL8JAAdHTryocuHVDDLwYJN5a2iAkkaKpAoGQmMeMzQFWcJvReLEFCrfAnaVLehTXUbKpUqlu16Hm7cyJ0pX2NQujR28+Zi8k4293KPvQ/+m+DCOgg7BQidL3q8FXOLdVfWsfbftSQkJ9C5UmeGOg/FrnjeBzDl9aYCSRoqkCiZkVJy+XY0+wIi2OsfweXb0QA4lDKlTXUb2la3wbVciSzXnzw5f56wESPQxD6hzI/TMGuTw92oo0LAz0c7/BV1FfSN/1v0WLl1ni96vPf0HosvLmb9v+vRSA3uVdwZ4jxE7eqopFKBJA0VSJScuPXwKfv8I9gXEMHxq/dJ0kisihnR8l1r2lQvzf8qW1HEKP1J8sSICMI+GUGcnx9Ww4dj9dGHOd/KV0q4dU4bUC5thCf3dLroMSI2gt8v/s7GoI0IBO+98x6DnAZRqmipPDuH8npSgSQNFUiUVxUdl8ihfyPZ6x/BoSt3eRyfhImhHv+rXIq21W3SnVfRxMdzZ/IUHm3eTLHWrSgz7Uf0i2Wyc2NmkhO1qe791r686NHJE6wq576TKW7H3Gah30I2B2/GUM+QnlV7MsBpACVNstjbRXljqUCShgokSl5ISNJwMvQ++/y1Q2C3H8UhBNROM6/ikDKvIqXkwR9/EPHTzxg72GM3bx5G5XO588KzRY9+6yDkMNpFj7VTFj12z7NFjzejb7LAbwHbQ7ZjrG/MB9U+wLuGt1qL8hYqFIFECNEe+AXQBxZJKae98L438DNwK+WluVLKRSnv/QR0AvSAvcBIKaUUQuwCbAED4AjwsZQyObN2qECi5LVn8yp7U4KKf3j68ypxJ44T9ulnANjNnIFpo0Z504Do29phL791cEc3ix5DHoWwwHcBu67twtTQlH7V+9Gneh+KGxXPgw4or4MCDyRCCH0gEGgDhAGngV5SSv80x3gDdaSUw18o2whtgGma8tJRYLyU8pAQwkxKGS20aVw3AD5SyrWZtUUFEkXXns2r7PWP4ETI8/Mq7UsmU37GFBJDrmL9xRhKennlOAtxpiL8tQElddFjMaiWsujRvmmuFz0GPghkvu989t/Yj5mRGf0d+9P73d4UNcz7FfpK4VIYAklDYIqU/9/encdXVZ37H/88mQMZICSMCWEWEAQEcQKlDhWHalsZ6lS9/hxQkFat13r1evvTVpzqULOMtr8AACAASURBVMGp1tpXxQG8zhPOIgpiZBAJMyQkJGSEhCQkOTnnuX/snXCIgRySnAzkeb9eeZlzzj5nr2UgX9Zaez9Lz3Ef3wGgqvP8jrmKhoPkZGA+MAlnI62lwBWqusHvmHDgdeBFVX31cG2xIDGtqWS/hy825fPJhvy6dZVu4uGe9f/L0M1pRJ1/Aan3/ZmQyCO7Z6VRPl/dTo+kv+Xc9BjT27npccxvmn3TY3pROgvWLGBp9lISohK4etTVzDhmBtFh0S3YCdOetIcgmQZMVdVr3MdXACf6h4YbJPOAApzRy82qmuW+9jBwDU6QzFfVO/3etwSYCHyAEzA2tWXapdp1lY/T8/h0fS6nr3yXKzZ+RHavgWTf/D+cfuqxdesqLcpTWe+mR0+L3fS4tmAtC1YvYHnuchKjE7lm9DVMHzbdqg4fhdpDkEwHzqkXJBNV9Sa/Y3oAZapaJSKzgBmqeoaIDMFZW5npHvoxcLuqLvV7bxSwEHhaVT9u4PzXAdcB9O/ff3xmZmZQ+mlMoGrXVda+/BYj//UIFSER3HvilXiOOZazjuB+lSNWUQzrX3dCJetb6m56HHspjLwIIpp2Rdn3ed8zf/V80vLS6N21N9cddx2/HPJLwkPCW7b9ps20hyBpdGqr3vGhQLGqxovIbUCUqt7rvnY3UKmqD9Z7z5XACfWnxuqzEYlpb6q2bGHHrBvx5u3m/TOv4Jmuo+rWVc4c7lwBNmloIlHhLVzUsXi7U5l47cvO9xExcOwvYezl0P+kI576UlW+3f0t81fPZ23BWvrF9GPWmFlcMOgCwkKOrEy/aX/aQ5CE4UxXnYlzVdZ3wKWqut7vmD6qmut+/yucUcdJIjITuBaYijO19SHwGPA5EKuque7nLwS+UtX5h2uLBYlpj7x797Lrllsp/+Ybus68hNW/uJKPNxfx5aaCuvtVJg9N4uyRvThzeE96HGEdsMNShZ0rYM2LsP5NqC6DhMHOKGXMJRDf7wg/Tvlq11csWLOA9KJ0BsQNYNaYWUwdMJXQIFQ4Nq2jzYPEbcR5OAEQCjyvqn8RkXuANFV9W0TmARcCNUAxcIOqbnRHJ0/iXLWlwIeqeouI9ALeBSLdz/wMZ12l5nDtsCAx7ZXW1JD/10co/uc/6TJxIv0eexRfXLe6dZWP0/PILakkRGB8anfOGnHw/SotoqoMNrwNqxc6FYolBAafAWMvcy4lDg+8zL6q8lnWZyxYs4Ate7YwOH4wN469kbNSz7Ly9R1QuwiS9sKCxLR3JW+9Re5/301YYqJT9HHECODAuspH6Xl84ne/yuCkrpw9sjdnj+zFuJRuhLTUukrxdljzEqx5GUqzIaqbszg/7jLoMzbgqS+f+vgo8yOeXPMkO0p2cEz3Y5g9djZTUqa07KXPJqgsSPxYkJiOYP+6dWTPuQlvaSl9591H3NSpPzkme0+Fc7/Khjy+3V4cvHUVnxd2fOmMUja8A94q6HksjLvcufIrwLvovT4v7+94n6fXPs3OfTsZ1WMUs8fN5tS+p1qgdAAWJH4sSExHUVNQQPbc37F/9Wp6XH89Sb+be8iij7X3q3ycnle3rhIdHsrkoYmc1ZLrKvv3OHfRr14IOasgJAyGTXVCZcjZENr4onqNr4Z3tr3D02ufJqc8h3E9xzFn7Bwm9pnY/PaZoLEg8WNBYjoSX3U1effey97FrxEzZQp9H3qQ0NjDlyWprvGxYruzrvLJhoPXVZw6YL0ZmNjEwpH+8jfA6hedmx7LC6BrTxgz07nqq+fwRt/u8Xp4Y+sbPPPDM+RX5DOx90TmjJvDuJ7jmt820+IsSPxYkJiORlXZ8/LL5N03j4iUFJKfXEDkwIEBv7d2XeXj9Dw2uOsqQ3rG1C3WN3tdxeuBLR/DmoXOjY++Gug3wVlLGXVxoxtyVXmrWLxpMc+te46iyiJO7Xsqs8fOZnTS6Ka3ybQ4CxI/FiSmoyr/diW7fv97tKaGfn99mJjTTmv8TfVkFVfw6Yb66yqRnDY0kUnuV8/YwK/M+omyAmeEsmYh5KdDWBSM+IVz1dfA0+Ew+7Hsr9nPKxtf4fkfn2dv1V6mJE9h9rjZDE9ofHRjgs+CxI8FienIPLt2kTXnJqo2biTplpvpcc01TV6oLqnw8MVmpw7Y11sLKS6vBmB471gmD01k8tAkJg5MaNqCvSrkrHYCZd1iqCyB+BTnvpSxl0LCoUdU5Z5yFm5YyAvrX2Bf9T7OTj2bG8fcyJDuLbffijlyFiR+LEhMR+erqCDnzjvZ98GHxJ13Hn3+8mdCoptXLNHnU9JzS1m6pYBlWwpJy9hDtddHRFgIEwckMGloIpOHJjKid9yRT4N5KmHTe84C/bbPAIUBk51RysgLD1mWpbS6lH+n/5t/p/+bCk8FUwdO5cYxNzIgfkCz+mqaxoLEjwWJORqoKkXP/p2Cxx4jcsRwUubPJ7xv3xb7/IrqGlbuKOarLYUs21LIprx9ACTGRHDqEGe0MnloIr3ijnAarCTbKcmy5iW3LEusU5Zl3OWQcmKD96bsrdzLC+tf4KWNL1HlreKCQRcwa8wsUmJTWqKrJkAWJH4sSMzRZN8XX5Dzh9uQiAiSH3+MLiecEJTz5JVWuqFSwLKthRSWOdNgw3rFMGlIEpOHJXLiwAS6RARYU0sVdi53Rinr3wBPOfQY4oxSxlwCcX1+8pai/UU8/+PzvLrpVbw+LxcNuYjrj7uePjE/Pda0PAsSPxYk5mhTtX0H2TfeSHV2Nr3vupPuv/lNUM/n8ykbd+/jKzdUvt1RTHWNj4jQEMandmfysEQmD0ni2L4BToNVlUH6m06o7PzGLctypnPV1zHnQdjB97/kV+Tz3LrneG3zawBMGzaNa0dfS1KXpGB017gsSPxYkJijkbe0lF233Ub5l0vpNmMGve+6E4lonT1BKj1eVu4oZtnWQpZuLmDjbmcaLKFrBKcM7sFpQ5OYNDSRvt0CWMcp2uZMe6192dnhMbq7W5blcugz5qBDc8tyeeaHZ3hr61uEhoQy85iZXD3qanpE9whGNzs9CxI/FiTmaKVeLwWP/42iZ58l+vjjSf7b44QlBla+pCXl76vk662FfLXF+SrYVwU4NcFq11ZOGtSDrpGHmQbzeWH7F85VXxvedcqy9BrtjFJGz4CuB8Iia18WT699mne3v0tkaCSXDr+Uq469im5R3YLc087FgsSPBYk52pW89x65d95FaLduJD/xBNGjR7VZW1SVTXn7WLalkKVbClm5o4hKj4/wUGFc/+6c5l5mPKpf/KE38dq/x9k3Zc1C55LikHA4ZqpzB/2Qs+rKsuwo2cFTa5/iwx0f0iW8C1eMvIIrRl5BXERcK/b46GVB4seCxHQGlenpZM2Zg7eomD5/vpf4X/yirZsEONNg32fucUcrBazPce60j48OZ9KQxLrLjJO7d2n4A/LSnUBZ+wpUFEJML2cP+rGXQ9IwALbs2cKTa57kk52fEBsRy38c+x9cNuIyuoQf4jNNQCxI/FiQmM6ipriYXXN/R0VaGglXX03PW29BQtvXxlKFZVV8vdW5xPirLYXsLq0EYGBi17qbIk8alEBsVL0te70e2LzELcuyBNQLySc4V32N+jVExbOhaAML1izgy+wv6R7ZnatHXc3M4TOJDmvePTedlQWJHwsS05mox0PevHnseelluk6aRL+/Pkxo/OFrX7UVVWVrflndaGXF9mL2e7yEhgjH9+9Wd5nxcf3iCQv1K7VSlu+UZVm9EAo2QFi0c6Pj2MtgwGR+KPqRBWsW8E3ONyRGJ3LN6GuYNmwakaEtuMtkJ2BB4seCxHRGexYtYve9fya8bx9SFiwgckj7LzdSVeNlVeZelm0t4KsthazbVYIqxEaFcergxLrLjPv3cKesVJ3S9qsXOmsqVSXQrT+MuRTGXsqq6iLmr5nPd7u/IyEqgREJIxgQP4ABcQPq/turSy/bG+UQLEj8WJCYzqpi1Sqy5/4O3b+fvg89SOwZZ7R1k47InvJqvt5WyFebnRFLTokzDZbaowuT3LvtTx7cg/jocPDsh43vOWXut39BXVmWcVfwbfc+vJ7xHjtKdpBRmsH+mv1154gOiyY1LpXUuNS6gBkYN5DUuFRiIlpwS+MOyILEjwWJ6cw8u3eTPecmKn/8kcS5N5E4a9YhN8tqz1SV7YXlztrK5nzStuTh2V9FpNYwpncXTkqOZULfGIYlRBK6Zxe68SN8mz9BS/LRkC7IgAnEnnsxDJpEfng4GaUZZJZm1oVLRkkGOeU5+NRXd87E6MSDRi+13/eL6UdYSIB39HdgFiR+LEhMZ+errCT37rspffsdYn/+c/rOu4+Qrke20ZWqgseDr9qDVleh1dXOV1UVvupqtMp97HGe0+rqg5+vbug591hPNb76z7mfVfd8ld85PZ4m/X+IiK2h59gSYkb2RFJPhf4nQeopkDgMRKj2VpO1L4uMkgx2lO4gszSTjJIMMkoz2Fu1t+5zwiSM5Njkg0YvtWGTEJVw1EyVWZD4sSAxxgmC4n++QP7DDxMxYABRI0a4v9jdX9D1fpH7PNVoteegX+C0xO8LESQyEomIQCIjCAmPOPC49rmICCTi4Ockot7zkZFIRLjzfGQk+zWUzXuqWFewn7W7y8mp8OEJDSOhW1fGDOrJqeGl9H35GTxZOXQZFEuvMcVERRY4berSA/qf7ARL/1Ogz3EQevBVY3sr9zojF3f0klmaWTeq8fgOBFtsROxBo5faKbPUuFSiwpqx70sbsCDxY0FizAFly74m/4H78VVXE3LQL2u/X8wH/bKOQCLCCan7vqHnDjwfEhlR7zP9QyACwsKC/i92VSWzqIKvtjiL9su3FbGvqobuEcItlesZ//lrSHkZ3c4/m6TzjiGsZB1kfgN7djgfEN7FubQ49RQnXJJPOGTpe6/PS255bl3A+IdNXkVe3XGC0Kdrn4NGL7Vh07trb0Kk/U03WpD4sSAxpnOr8fpYsb2Yxd9n8cGPu4moKGNO9pdMXv8FoVFRJM66noTf/paQqmKnQvHOFU4xyd0/AgohYU7dr/4nH/jq2nh9rwpPBZmlmc5aTOmOuqDJLM2k3FNed1xkaCT94/rXhcvA+APTZW15l367CBIRmQo8DoQCz6nq/fVevwp4CNjlPjVfVZ9zX3sQOB8IAT4GfgdEA4uBwYAXeEdV/9hYOyxIjDG1Sio8vP1DDovTsijasIVr09/lxNx0anr1IeX224g/d+qBEVNlCWStdMIlczns+t6pAQaQeAyk+gVLt/4N7q3SEFWlcH/hQaOX2v/uKtuFV711xyZEJfxkwT81PpWU2BTCQ8IPc5bma/MgEZFQYDNwNpANfAdcoqrpfsdcBUxQ1Tn13nsKTsDUblC9DLgDWAmcqKqfi0gE8Clwn6p+cLi2WJAYYxqycXcpi9Oy2fz+Z1yS9joDS3MpGjSCPv91B4MnNbDPi6fSqf218xt31PKtc+8KQFw/J1BqwyVpxGH3qz8Uj9dDVlnWQeswtUFTXFlcd1yohDoL/u76i3/QJEYntsj0YaBBEszr1yYCW1V1u9ugV4CLgPTDvsuhQBQQAQgQDuSpagXwOYCqVovIKiA5CG03xnQCw3vH8d8XjKR66nA+W/9LlvzzJSZ+vpjqa37LyyNOIeqGOfz89NEHqhaHRzlBkXqy89jnhfx0Z7Syczlkfg0/OnumENXNXbx3g6XvOAhrvMx/eGg4g+IHMSh+0E9eK6kq+Um4ZJRmsCJ3BVW1IyUgJjymLlzuOvGuoN8PE8wRyTRgqqpe4z6+Amc0McfvmKuAeUABzujlZlXNcl97GLgGJ0jmq+qd9T6/G7AKOKs2rA7FRiTGmEDtzilk7QOP0+fjN/GK8ObwM/FcfCm/PmUw41O7H/5f+qqwJ8NdZ3Gnw4q2OK+FRUG/CQdGLCkTITK2RdrsUx+7y3fXXbZcO5rJKc/hrYucvVuaoj1MbU0HzqkXJBNV9Sa/Y3oAZapaJSKzgBmqeoaIDMFZW5npHvoxcLuqLnXfFwa8AyxR1ccOcf7rgOsA+vfvPz4zMzMo/TTGHJ2qdu5k8z33E7bscwqju/H8yHPZOXYS005I5eLj+9Ez0L3rywoOXsDP/cEpOCkh0Hu0c7lxbbjE9Axup45QewiSk4E/qeo57uM7AFR13iGODwWKVTVeRG4DolT1Xve1u4FKVX3Qffw8TgDNDaQtNiIxxjRVRVoauffNozo9nexeA3lk2PlsThrIlGFJTJ+QzBnDexERdgRrIVX7IPs7J1gyv4HsNKgt2ZIw2A0V97LjhEEBL+AHQ3sIkjCc6aozca7K+g64VFXX+x3TR1Vz3e9/hTPqOElEZgLXAlNxprY+BB5T1XdE5M/ACGC6ql8tg8OwIDHGNIf6fJS89TYFjz5KTX4+u8ZN4pGBZ5Pu60pC1wh+Na4fMyakcEzvJkxV1VRD7toD02E7lzsbewHE9D5w933/k6HXsdDEaaqmaPMgcRtxHvAYzuW/z6vqX0TkHiBNVd8WkXnAhUANUAzcoKob3dHJkzhXbSnwoareIiLJQBawEahdWaq7ZPhQLEiMMS3BV1FB0XP/oOj558HnY9+F03lx0BTe31aKx6uMSY5n2oQULhzT1ykk2aST+KBw04E1lp3LoSTLeS0yzllb6X+yEy59j3cuAAiSdhEk7YUFiTGmJXlyc8l/9FFK336H0MREut4wmyUpE1i0KoeNu/cRGRbC1FG9mTEhhZMH9SDkUFsKB2pv1sEL+AUbnOdDI5wwqZ0OS5kI0S23b70FiR8LEmNMMOz/4Qfy7pvH/jVriBwxgp63/ycZKSNYlJbFm6t3UVpZQ79u0UyfkMzFxyeTktBCW/9WFLuL92645KwGXw0gzvRX3f0sp0BcnyafxoLEjwWJMSZYVJV9H3xA3sMPU5OTS8xZZ9Lrttvw9U3mo/Q8FqdlsWxrIapw6pAezJiQwjnH9iYqvAXXOqorYFfagQX8rJVQW4Lltu0BlXNpiAWJHwsSY0yw+SorKX7hXxQ9+yw+j4eEyy4j8cYbCI2LY9fe/fzv99ks/j6LrOL9xEaFcdHYvkwfn8JxyfEtX8TSWwO7f4C8H+H43zb5YyxI/FiQGGNaiyc/n4LHH6fk9TcIjY8nce5NdJ8xAwkLw+dTVuwoYnFaNu+vy6WqxscxvWKZPiGZX43rR4+Y9rWnvAWJHwsSY0xrq0xPJ+/+B6hYuZKIIYPpdfvtxEyeXPd6aaWHd9fmsigtizVZewkPFc4c3osZJyRz2tAkwkLbvqy8BYkfCxJjTFtQVco+/ZS8Bx/Cs3MnXU+bTK/bbydy8OCDjtuct4/FaVm8vmoXReXV9IyN5OLxyUwfn8ygpLbbN96CxI8FiTGmLfmqq9nz4kIKn3oKX0UF3WfOJPGmOYR1737QcR6vj8835rMoLZvPN+Xj9SkTUrszY0IK5x3Xh5jI1t0n3oLEjwWJMaY9qCkupnD+fPa8uoiQrl1JvOEGEi671Nk5sp78fZW8sWoXi9Ky2FZQTpeIUM4f3YfpE1I4YUAjxSNbiAWJHwsSY0x7UrVlC3kPPEj5smWEp/an13/+JzFnnNFgOKgqq3bu5bXvs3hnbS5lVTUMTOzKtPHOvSm94+3O9lZhQWKMaY/Kli4l74EHqd62jS4nnkivO/5I1PDhhzy+orqGD9btZlFaFt/uKCZE4PRhSUyfkMKZI3oSGdaydbgsSPxYkBhj2iv1eNjz6iIKn3gCb2kp3aZdTNLcuYQlJR32fZlF5bz2fTavfZ9Nbkkl3buE80u3eOSIPi2zz7sFiR8LEmNMe+ctKaHwyacoXriQkIgIelx/PQlXXUlI5OHvLfH6lGVbC1mUlsXH6/Oo9voY3S+eGROSuXBMP+K7NH1fdwsSPxYkxpiOojojg7yHHqbs008J79ePnn+4ldipUwNaXN9TXs3ba3NYlJbF+pxSIsJCWHb7z+gZ27R1FAsSPxYkxpiOpnzFCvLm3U/Vpk1EH388ve74I9GjRwf8/h93lbB8WxHXnvbTvd8DZUHix4LEGNMRqdfL3tdfp+Cxx/EWFRF34S/oecsthPfu3SrnDzRI2v4efGOMMQ2S0FC6T5/O4CUf0uPaa9n34RK2TT2Xgifm46uoaOvm1bEgMcaYdi40Joaet97CoPffI+ZnUyhcsIBt555HyVtvob6AdhwPKgsSY4zpICKSk0l+9FFSF75IWFISObf/kYwZM6lYtapN22VBYowxHUyX8eMZsOhV+j5wPzUFBWReehnZv7+Z6uzsNmmPBYkxxnRAEhJC/EUXMfiD90mcPZuyL75g+3nnk//XR/CWlbVqWyxIjDGmAwvp0oWkm+Yw+MMPiDt3KkV//zvbzpnKnkWLUK+3ddrQKmcxxhgTVOG9e9P3gQcYsOhVIlJT2X33/7Dj1xfjycsP+rktSIwx5igSfdxxpC58kX6PPkJE/xTCkhKDfs6gBomITBWRTSKyVUT+2MDrV4lIgYiscb+u8XvtQRFZLyIbRORv4tYHEJG/iEiWiLTuJKAxxnQQIkLcueeS/MQTSEjwxwtBO4OIhAILgHOBkcAlIjKygUNfVdWx7tdz7ntPAU4FjgNGAScAp7vHvwNMDFa7jTHGHJlgRtVEYKuqblfVauAV4KIA36tAFBABRALhQB6Aqq5Q1dwgtNcYY0wTBDNI+gFZfo+z3efqu1hEfhCR10QkBUBVlwOfA7nu1xJV3XAkJxeR60QkTUTSCgoKmtYDY4wxjQpmkDRU87h+hch3gAGqehzwCfAvABEZAowAknHC5wwROe1ITq6qz6rqBFWdkNTIBjHGGGOaLphBkg2k+D1OBnL8D1DVIlWtch/+HRjvfv8rYIWqlqlqGfABcFIQ22qMMaaJghkk3wFDRWSgiEQAvwHe9j9ARPr4PbwQqJ2+2gmcLiJhIhKOs9B+RFNbxhhjWkfQgkRVa4A5wBKcEFikqutF5B4RudA9bK57ie9aYC5wlfv8a8A2YB2wFlirqu9A3WXB2UAXEckWkT8Fqw/GGGMaZxtbGWOMaZDtkOhHRAqAzCa+PREobMHmdATW586hs/W5s/UXmt/nVFVt9GqlThEkzSEiaYEk8tHE+tw5dLY+d7b+Quv12WptGWOMaRYLEmOMMc1iQdK4Z9u6AW3A+tw5dLY+d7b+Qiv12dZIjDHGNIuNSIwxxjSLBYkrgL1TZonIOnfflGWHKInfoTTWZ7/jpomIikiHvuKlOfvjdFSB/IxFZIaIpLs3B7/U2m1saQH8nB/1+xlvFpG9bdHOlhRAn/uLyOcistotknteizZAVTv9FxCKcyf9IJzS9WuBkfWOifP7/kLgw7Zud7D77B4XCywFVgAT2rrdQf4ZXwXMb+u2tnKfhwKrge7u455t3e5g97ne8TcBz7d1u1vh5/wscIP7/UggoyXbYCMSR6N7p6hqqd/Drvy0knFHE+h+MfcCDwKVrdm4IGjO/jgdVSB9vhZYoKp7AFQ1+Bt8B9eR/pwvAV5ulZYFTyB9ViDO/T6eegV0m8uCxBHQ3ikiMltEtuH8Yp3bSm0Llkb7LCLjgBRVfbc1GxYkTd4fpwMLpM/DgGEi8rWIrBCRqa3WuuAI9OeMiKQCA4HPWqFdwRRIn/8EXO7WKXwfZyTWYixIHIHsnYKqLlDVwcDtwF1Bb1VwHbbPIhICPArc2motCq4m74/TgQXS5zCc6a0pOP86f05EugW5XcEU0N9l12+A11TVG8T2tIZA+nwJ8IKqJgPnAf92/463CAsSR6N7p9TzCvDLoLYo+BrrcywwCvhCRDJw9oN5uwMvuDdnf5yOKpA/19nAW6rqUdUdwCacYOmojuTv8m/o+NNaEFif/x+wCOp2oI3CqcPVIixIHIHsneL/l+t8YEsrti8YDttnVS1R1URVHaCqA3AW2y9U1Y5aRrk5++N0VI32GXgT+BmAiCTiTHVtb9VWtqxA+oyIHAN0B5a3cvuCIZA+7wTOBBCREThB0mJ7kIe11Ad1ZKpaIyK1e6eE4lzFsV5E7gHSVPVtYI6InAV4gD3AlW3X4uYLsM9HjQD7O9fdK6cGKObA/jgdUoB9XgL8XETSAS9wm6oWtV2rm+cI/lxfAryi7mVMHVmAfb4V+LuI3Iwz7XVVS/bd7mw3xhjTLDa1ZYwxplksSIwxxjSLBYkxxphmsSAxxhjTLBYkxhhjmsWCxJhmEpE/icgf2kE7Mtx7QYxpVRYkxhhjmsWCxJgGiEhXEXlPRNaKyI8iMtP/X/wiMkFEvvB7yxgR+UxEtojIte4xfURkqbvvxY8iMtl9/ikRSXP3//j/fufMEJH7RGS5+/rxIrJERLaJyCz3mCnuZ77h7iHydEM1k0TkchFZ6Z77GREJDeb/L9O5WZAY07CpQI6qjlHVUcCHjRx/HE7pnJOBu0WkL3ApsERVxwJjgDXusXeq6gT3PaeLyHF+n5OlqicDXwEvANNw6pzd43fMRJw7lUcDg4Ff+zfELYExEzjVPbcXuOwI+m7MEbESKcY0bB3wsIg8ALyrql+JNFRktc5bqrof2C8in+P8sv8OeF5EwoE3VbU2SGaIyHU4f//64Gw09IP7Wm0Jj3VAjKruA/aJSKVfVd6VqrodQEReBiYBr/m15UycgpPfuW2OBjr6PiOmHbMgMaYBqrpZRMbjlNyeJyIf4dTgqh3FR9V/y08/QpeKyGk4I5V/i8hDOCONPwAnqOoeEXmh3mfVVh/2+X1f+7j27+tPzlXvsQD/UtU7GummMS3CpraMaYA7NVWhqi8CDwPHAxkcKC1/cb23XCQiUSLSA2dvj+/cjZPyVfXvwD/cz4gDyoESEekFnNuE5k10K72G4ExhLav3+qfANBHp6fYlwW2LMUFhIxJjGjYaeEhEfDgVn2/AmSL6h4j8F/BtveNXAu8B/YF7VTVHRK4EbhMRD1AG/FZVd4jIqfPhZQAAAHZJREFUamA9Trn2r5vQtuXA/W4blwJv+L+oqukichfwkRs2HmA2kNmEcxnTKKv+a0wHIiJTgD+o6gVt3RZjatnUljHGmGaxEYkxxphmsRGJMcaYZrEgMcYY0ywWJMYYY5rFgsQYY0yzWJAYY4xpFgsSY4wxzfJ/cVKWuOLwQiYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# summarize results\n",
    "print(\"Best: %f using %s\" % (gsearch3_1.best_score_, gsearch3_1.best_params_))\n",
    "test_means = gsearch3_1.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = gsearch3_1.cv_results_[ 'std_test_score' ]\n",
    "train_means = gsearch3_1.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = gsearch3_1.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "pd.DataFrame(gsearch3_1.cv_results_).to_csv('my_preds_subsampleh_colsample_bytree_1.csv')\n",
    "\n",
    "# plot results\n",
    "test_scores = np.array(test_means).reshape(len(colsample_bytree), len(subsample))\n",
    "train_scores = np.array(train_means).reshape(len(colsample_bytree), len(subsample))\n",
    "\n",
    "for i, value in enumerate(colsample_bytree):\n",
    "    pyplot.plot(subsample, -test_scores[i], label= 'test_colsample_bytree:'   + str(value))\n",
    "#for i, value in enumerate(min_child_weight):\n",
    "#    pyplot.plot(max_depth, train_scores[i], label= 'train_min_child_weight:'   + str(value))\n",
    "    \n",
    "pyplot.legend()\n",
    "pyplot.xlabel( 'subsample' )                                                                                                      \n",
    "pyplot.ylabel( 'Log Loss' )\n",
    "pyplot.savefig( 'subsample_vs_colsample_bytree1.png' )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以得出，最优的参数colsample_bytree =  0.9, subsample = 0.8，此时logloss是0.581140，比上一轮的缺省参数logloss是0.586436好  \n",
    "同时也可以看出，这两个最优参数都在所设置候选参数的边界，其实应该再往右扩大边界，再进行调优的，但是电脑性能有限，跑太慢，暂时以本轮的最优参数作为下一轮调优的输入"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
